Question

I need a very basic key-value store for java. I started with a HashMap but it seems that HashMap is somewhat space inefficient (I’m storing ~20 million records, and seems to require ~6GB RAM).

The map is Map<Integer,String>, and so I’m considering using GNU Trove TIntObjectHashMap<byte[]>, and storing the map value as an ascii byte array rather than String.

As an alternative to that, is there a key-value store that only requires adding jar files, does not hold the entire map in RAM at once, and is still reasonably fast?

Answer 1

Use Berkeley DB.

Berkeley DB stores object graphs, objects in collections, or simple binary key/value data directly in an a btree on disk. This simple, highly efficient approach removes all the unnecessary overhead in ORM solutions. Using the Direct Persistence Layer (DPL) Java developers annotate classes with storage information, much like JPA. This approach is familiar, efficient, and fast. The DPL reduces the complexity of data storage while not sacrificing speed.

This should definitely give you huge gains in memory and speed, while not increasing the complexity of your application. Enjoy!