Say I have a very large dictionary / associative array and I want to be able to look up values quickly by key with a minimum of memory use – so storing the whole thing in memory won’t work.
If I were to create a full MySQL database with two columns I could query the database and rapidly get an answer even though the server process isn’t taking up gigabytes of memory space and isn’t churning on a multi-gigabyte disk read. Presumably there’s some magic indexing going on against the binary file. Obviously MySQL is overkill.
Is there some obvious mature technology that I’m missing which can fill the gap? Preferably with a dedicated server process that runs in the background? I’m not interested in writing a data structure from scratch.
Well, you could try Solr, which is based on Lucene.
Another interesting alternative could be a key/value database, like Berkeley DB for example, which is also quite nice.