Question

What ways do you know to evaluate the efficiency of a hash function besides generating a large set of values and see the distribution of values?
By efficiency I mean that the keys generated by your hash function distribute evenly. Is there a way to prove this without actually testing for actual values?

Answer 1

A hash function is only even in the context of the data being hashed

Consider two data sets:

Set 1

1, 3, 6, 2, 7, 9, 5, 8, 4

Set 2

65355, 96424664, 86463624, 133, 643564,  24232, 88677, 865747, 2224

A good hashing function for one set (ie mod 10 for set 1) gives no collisions and could be seen as the perfect hash for that data set

However apply it to the second set and there are collisions everywhere

Hash = (x * 37) mod 256

Is much better for the second set but may not suit the first set quite so well… Especially when partitioning the hash for eg a small number of buckets.

What you can do is evaluate a hash against random data that you “expect” your function to have to handle… But that is making assumptions…

Premature optimisation is looking for the perfect hash function before you have enough real data to base your assessment on.

You should get enough data well before the cost of rehashing becomes prohibitive to change your hash function

Update

Lets suppose we are looking for a hash function that generates an 8 bit hash of the input data. Lets further suppose that the hash function is supposed to take byte-streams of varying length.

If we assume that the bytes in the byte-streams are uniformly distributed, we can make some assessment of different hash functions.

int hash = 0;
for (byte b in datastream) hash = hash xor b;

This function will produce uniformly distributed hash values for the specified data set, and would therefore be a good hash function in this context. If you don’t see why this is, then you might have other problems.

int hash = 37;
for (byte b in datastream hash = (31 * hash + b) mod 256;

This function will produce uniformly distributed hash values for the specified data set, and would therefore be a good hash function in this context.

Now lets change the data set from being variable length strings of random numbers in the range 0 to 255 to being variable length strings comprising English sentences encoded as US-ASCII.

The XOR is then a poor hash because the input data never has the 8th bit set and as a result only generates hashes in the range 0-127, also there is a higher likelyhood of some “hot” values because of the letter frequency in english words and the cancelling affect of the XOR.

The pair of primes remains reasonably good as a hash function because it uses the full output range and the prime initial offset coupled with a different prime multiplier tends to spread the values out. But it is still weak for collisions due to how English language is structured… Something that only testing with real data can show.