I had interview yesterday, and I had such question – there is server and database, which store big text information(every text can be >10 Mb). Any query to database is expensive. You need to design a program(server) which will take a text and say – is this text in database or not.
I request such decision –
Make a good hash of every text. Then make a set of all hashes. If hash of input text is not in set – so there is no such text, else – go to database and try to find this text by hash and equal deeper. Minus of such decision that it can be a lot of texts in database, so set will be a bit bigger=)
May be your know how to solve it right?
As discussed in the comments, the basic concept is to reduce the number of queries to the database by eliminating query texts that cannot possibly be in the database.
As you suggest, hashing is an excellent way to do this – if you use a good hash function (i.e. low probability of collision) then you would never need to actually compare the texts, since you would (almost certainly) never get a false positive.
A variant of hashing (if memory becomes an issue) is to use a Bloom Filter: “a space-efficient probabilistic data structure that is used to test whether an element is a member of a set.” This approach uses far less memory than plain hashing, at the expense of some false positives (i.e. occasionally you would need to compare a text with the database to make sure whether you had a match). Bloom filters let you adjust the tradoff between memory and false positive rate. They are used in NoSQL databases such as BigTable and Apache Cassandra.
Update as David Schwartz rightly points out, the hash function does matter, particularly in the case where a significant propertion of the queries are in the database – see our discussion in the comments below. Same issue applies to Bloom filters which are ideal when you want to narrow down your disk access to a small subset of files (for example) but not for eliminating disk/database access entirely, because of the false positives. I originally mentioned MurmurHash3 by way of a speed benchmark, but because it is non-cryptographic, it is unsuitable for this question. When I actually implemented approaches like this in the past, I used SHA1, which is cryptographic.