Question

I have some configuration data that I’d like to model in code as so:

Key1,  Key2,  Key3,  Value null,  null,  null,  1 1,     null,  null,  2 9,     null,  null,  21 1,     null,  3,     3 null,  2,     3,     4 1,     2,     3,     5 

With this configuration set, I then need to do lookups on bazillion (give or take) {Key1, Key2, Key3} tuples to get the ‘effective’ value. The effective value to use is based on a Key/Priority sum whereby in this example:

Key1 - Priority 10 Key2 - Priority 7 Key3 - Priority 5 

So a specifc query which has a config entry on Key1=null, Key2=match, and Key3=match beats out one that has Key1=match, Key2=null, and Key3=null since Key2+Key3 priority > Key1 priority… That make sense?!

given a key of {1, 2, 3} the value should be 5. given a key of {3, 2, 3} the value should be 4. given a key of {8, 10, 11} the value should be 1. given a key of {1, 10, 11} the value should be 2. given a key of {9, 2, 3} the value should be 4. given a key of {8, 2, 3} the value should be 4. given a key of {9, 3, 3} the value should be 21. 

Is there an easy way to model this data structure and lookup algorithm that is generic in that the # and types of keys is variable, and the ‘truth table’ (the ordering of the lookups) can be defined dynamically? The types being generics instead of ints would be perfect (floats, doubles, ushorts, etc), and making it easy to expand to n number of keys is important too!

Estimated ‘config’ table size: 1,000 rows, Estimated ‘data’ being looked up: 1e14

That gives an idea about the type of performance that would be expected.

I’m looking for ideas in C# or something that can translate to C# easily.

Answer 1

I’m assuming that there are few rules, and a large number of items that you’re going to check against the rules. In this case, it might be worth the expense of memory and up-front time to pre-compute a structure that would help you find the object faster.

The basic idea for this structure would be a tree such that at depth i, you would follow the ith element of the rule, or the null branch if it’s not found in the dictionary.

To build the tree, I would build it recursively. Start with the root node containing all possible rules in its pool. The process:

• Define the current value of each rule in the pool as the score of the current rule given the path taken to get to the node, or -infinity if it is impossible to take the path. For example, if the current node is at the ‘1’ branch of the root, then the rule {null, null, null, 1} would have a score of 0, and the rule {1, null, null, 2} would have a score 10
• Define the maximal value of each rule in the pool as its current score, plus the remaining keys’ score. For example, if the current node is at the ‘1’ branch of the root, then the rule {null, 1, 2, 1} would have a score of 12 (0 + 7 + 5), and the rule {1, null, null, 2} would have a score 10 (10 + 0 + 0).
• Remove the elements from the pool that have a maximal value lower than the highest current value in the pool
• If there is only one rule, then make a leaf with the rule.
• If there are multiple rules left in the pool, and there are no more keys then ??? (this isn’t clear from the problem description. I’m assuming taking the highest one)
• For each unique value of the (i+1)th key in the current pool, and null, construct a new tree from the current node using the current pool.

As a final optimization check, I would check if all children of a node are leaves, and if they all contain the same rule, then make the node a leaf with that value.

given the following rules:

null,  null,  null = 1 1,     null,  null = 2 9,     null,  null = 21 1,     null,  3    = 3 null,  2,     3    = 4 1,     2,     3    = 5 

an example tree:

       key1   key2   key3 root:  |----- 1  |      |----- 2          = 5  |      |-----null  |             |----- 3   = 3  |             |-----null = 2  |----- 9  |      |----- 2  |      |      |----- 3   = 4  |      |      |-----null = 21  |      |-----null        = 21  |-----null         |----- 2          = 4         |-----null        = 1 

If you build the tree up in this fashion, starting from the highest value key first, then you can possibly prune out a lot of checks against later keys.

Edit to add code:

class Program {     static void Main(string[] args)     {         Config config = new Config(10, 7, 5)         {             { new int?[]{null,  null,  null},  1},             { new int?[]{1,     null,  null},  2},             { new int?[]{9,     null,  null},  21},             { new int?[]{1,     null,  3},     3 },             { new int?[]{null,  2,     3},     4 },             { new int?[]{1,     2,     3},     5 }         };          Console.WriteLine('5 == {0}', config[1, 2, 3]);         Console.WriteLine('4 == {0}', config[3, 2, 3]);         Console.WriteLine('1 == {0}', config[8, 10, 11]);         Console.WriteLine('2 == {0}', config[1, 10, 11]);         Console.WriteLine('4 == {0}', config[9, 2, 3]);         Console.WriteLine('21 == {0}', config[9, 3, 3]);                     Console.ReadKey();     } }   public class Config : IEnumerable {     private readonly int[] priorities;     private readonly List<KeyValuePair<int?[], int>> rules =         new List<KeyValuePair<int?[], int>>();     private DefaultMapNode rootNode = null;      public Config(params int[] priorities)     {         // In production code, copy the array to prevent tampering         this.priorities = priorities;     }      public int this[params int[] keys]     {         get         {             if (keys.Length != priorities.Length)             {                 throw new ArgumentException('Invalid entry - wrong number of keys');             }              if (rootNode == null)             {                 rootNode = BuildTree();                 //rootNode.PrintTree(0);             }              DefaultMapNode curNode = rootNode;             for (int i = 0; i < keys.Length; i++)             {                 // if we're at a leaf, then we're done                 if (curNode.value != null)                     return (int)curNode.value;                  if (curNode.children.ContainsKey(keys[i]))                     curNode = curNode.children[keys[i]];                 else                     curNode = curNode.defaultChild;             }              return (int)curNode.value;         }     }      private DefaultMapNode BuildTree()     {         return new DefaultMapNode(new int?[]{}, rules, priorities);     }      public void Add(int?[] keys, int value)     {         if (keys.Length != priorities.Length)         {             throw new ArgumentException('Invalid entry - wrong number of keys');         }         // Again, copy the array in production code         rules.Add(new KeyValuePair<int?[], int>(keys, value));          // reset the tree to know to regenerate it.         rootNode = null;     }      public IEnumerator GetEnumerator()     {         throw new NotSupportedException();     }  }   public class DefaultMapNode {     public Dictionary<int, DefaultMapNode> children = new Dictionary<int,DefaultMapNode>();     public DefaultMapNode defaultChild = null; // done this way to workaround dict not handling null     public int? value = null;      public DefaultMapNode(IList<int?> usedValues, IEnumerable<KeyValuePair<int?[], int>> pool, int[] priorities)     {         int bestScore = Int32.MinValue;          // get best current score         foreach (KeyValuePair<int?[], int> rule in pool)         {             int currentScore = GetCurrentScore(usedValues, priorities, rule);             bestScore = Math.Max(bestScore, currentScore);         }          // get pruned pool         List<KeyValuePair<int?[], int>> prunedPool = new List<KeyValuePair<int?[], int>>();         foreach (KeyValuePair<int?[], int> rule in pool)         {             int maxScore = GetCurrentScore(usedValues, priorities, rule);             if (maxScore == Int32.MinValue)                 continue;              for (int i = usedValues.Count; i < rule.Key.Length; i++)                 if (rule.Key[i] != null)                     maxScore += priorities[i];              if (maxScore >= bestScore)                 prunedPool.Add(rule);         }          // base optimization case, return leaf node         // base case, always return same answer         if ((prunedPool.Count == 1) || (usedValues.Count == prunedPool[0].Key.Length))         {             value = prunedPool[0].Value;             return;         }          // add null base case         AddChild(usedValues, priorities, prunedPool, null);         foreach (KeyValuePair<int?[], int> rule in pool)         {             int? branch = rule.Key[usedValues.Count];             if (branch != null && !children.ContainsKey((int)branch))             {                 AddChild(usedValues, priorities, prunedPool, branch);             }         }           // if all children are the same, then make a leaf         int? maybeOnlyValue = null;         foreach (int v in GetAllValues())         {             if (maybeOnlyValue != null && v != maybeOnlyValue)                 return;             maybeOnlyValue = v;         }         if (maybeOnlyValue != null)             value = maybeOnlyValue;      }      private static int GetCurrentScore(IList<int?> usedValues, int[] priorities, KeyValuePair<int?[], int> rule)     {         int currentScore = 0;         for (int i = 0; i < usedValues.Count; i++)         {             if (rule.Key[i] != null)             {                 if (rule.Key[i] == usedValues[i])                     currentScore += priorities[i];                 else                     return Int32.MinValue;             }         }         return currentScore;     }      private void AddChild(IList<int?> usedValues, int[] priorities, List<KeyValuePair<int?[], int>> prunedPool, Nullable<int> nextValue)     {         List<int?> chainedValues = new List<int?>();         chainedValues.AddRange(usedValues);         chainedValues.Add(nextValue);                     DefaultMapNode node = new DefaultMapNode(chainedValues, prunedPool, priorities);         if (nextValue == null)             defaultChild = node;         else             children[(int)nextValue] = node;     }      public IEnumerable<int> GetAllValues()     {         foreach (DefaultMapNode child in children.Values)             foreach (int v in child.GetAllValues())                 yield return v;         if (defaultChild != null)             foreach (int v in defaultChild.GetAllValues())                 yield return v;         if (value != null)             yield return (int)value;     }      public void PrintTree(int depth)     {         if (value == null)             Console.WriteLine();         else         {             Console.WriteLine(' = {0}', (int)value);             return;         }          foreach (KeyValuePair<int, DefaultMapNode> child in children)         {             for (int i=0; i<depth; i++)                 Console.Write('    ');             Console.Write(' {0}  ', child.Key);                             child.Value.PrintTree(depth + 1);         }         for (int i = 0; i < depth; i++)             Console.Write('    ');         Console.Write('null');         defaultChild.PrintTree(depth + 1);     } }