Home/ Questions/Q 6973811
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T17:09:24+00:00

I often keep track of duplicates with something like this: processed = set() for

  • 0

I often keep track of duplicates with something like this:

processed = set() 
for big_string in strings_generator:
    if big_string not in processed:
        processed.add(big_string)
        process(big_string)

I am dealing with massive amounts of data so don’t want to maintain the processed set in memory. I have a version that uses sqlite to store the data on disk, but then this process runs much slower.

To cut down on memory use what do you think of using hashes like this:

processed = set() 
for big_string in string_generator:
    key = hash(big_string)
    if key not in ignored:
        processed.add(key)
        process(big_string)

The drawback is I could lose data through occasional hash collisions.
1 collision in 1 billion hashes would not be a problem for my use.

I tried the md5 hash but found generating the hashes became a bottleneck.

What would you suggest instead?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I’m going to assume you are hashing web pages. You have to hash at most 55 billion web pages (and that measure almost certainly overlooks some overlap).

    You are willing to accept a less than one in a billion chance of collision, which means that if we look at a hash function which number of collisions is close to what we would get if the hash was truly random[ˆ1], we want a hash range of size (55*10ˆ9)*10ˆ9. That is log2((55*10ˆ9)*10ˆ9) = 66 bits.

    [ˆ1]: since the hash can be considered to be chosen at random for this purpose,
    p(collision) = (occupied range)/(total range)

    Since there is a speed issue, but no real cryptographic concern, we can use a > 66-bits non-cryptographic hash with the nice collision distribution property outlined above.

    It looks like we are looking for the 128-bit version of the Murmur3 hash. People have been reporting speed increases upwards of 12x comparing Murmur3_128 to MD5 on a 64-bit machine. You can use this library to do your speed tests. See also this related answer, which:

    • shows speed test results in the range of python’s str_hash, which speed you have already deemed acceptable elsewhere – though python’s hash is a 32-bit hash leaving you only 2ˆ32/(10ˆ9) (that is only 4) values stored with a less than one in a billion chance of collision.
    • spawned a library of python bindings that you should be able to use directly.

    Finally, I hope to have outlined the reasoning that could allow you to compare with other functions of varied size should you feel the need for it (e.g. if you up your collision tolerance, if the size of your indexed set is smaller than the whole Internet, etc, …).

    • 0