Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Keys are long hashes so dataset won’t fit into the ram. It is not that big: 10-20Gb yet storing it in RAM is too costly for me. I have 100-1000 queries per second asking whether key exists and occasional 1-10 inserts. For a while I’ve been using mongo but it seems it is not up to the task since it lacks ability to control which indexes are stored into ram and which are not, so it hits hard disk quite often.
I would gladly use redis or something similar but ram is the bottleneck.
Hash your keys and put them into either a relational table or into a hash-table. That is the easiest solution. Next would be the use of Bloom filters which would be able to sort 99% or even more of the accesses out (in case the key is not present – in case it is there you don’t gain anything because it could be a false positive).
Not sure why your hashes are that long. What hashing technique are you using? You might want to switch to something like SHA-160 which has a small output yet is (almost) cryptographically secure.