Question

I was wondering: what are maximum number of bytes that can safely be hashed while maintaining the expected collision count of a hash function?

For md5, sha-*, maybe even crc32 or adler32.

Answer 1

Your question isn’t clear. By “maximum number of bytes” you mean “maximum number of items”? The size of the files being hashed has no relation with the number of collisions (assuming that all files are different, of course).

And what do you mean by “maintaining the expected collision count”? Taken literally, the answer is “infinite”, but after a certain number you will aways have collisions, as expected.

As for the answer to the question “How many items I can hash while maintaining the probability of a collision under x%?”, take a look at the following table:

http://en.wikipedia.org/wiki/Birthday_problem#Probability_table

From the link:

For comparison, 10^-18 to 10^-15 is the uncorrectable bit error rate of a typical hard disk [2]. In theory, MD5, 128 bits, should stay within that range until about 820 billion documents, even if its possible outputs are many more.

This assumes a hash function that outputs a uniform distribution. You may assume that, given enough items to be hashed and cryptographic hash functions (like md5 and sha) or good hashes (like Murmur3, Jenkins, City, and Spooky Hash).

And also assumes no malevolent adversary actively fabricating collisions. Then you really need a secure cryptographic hash function, like SHA-2.

And be careful: CRC and Adler are checksums, designed to detect data corruption, NOT minimizing expected collisions. They have proprieties like “detect all bit zeroing of sizes < X or > Y for inputs up to Z kbytes”, but not as good statistical proprieties.

EDIT: Don’t forget this is all about probabilities. It is entirely possible to hash only two files smaller than 0.5kb and get the same SHA-512, though it is extremely unlikely (no collision has ever been found for SHA hashes till this date, for example).