Question

Possible Duplicate:
Are they any decent on-disk implementations of Java’s Map?

I have a piece of code (that I didn’t write) that reads millions of CSV rows to a Map, then processes it.

I got to the point where I simply ran out of RAM

My options are

1. Rewrite the code, trying to stream the data, however since some calculations might need the entire data set (e.g. calculation that might need both the very first and very last row in the data set)

2. Write a Class that implements java.util.Map but will persist the data into a database

3. Simply rewrite the code and insert / select from a database directly, but I’d rather try #2 first

So the thought of a DB backed Map all of a sudden made sense to me, so before starting to write it, I wanted to ask if there is a well known pattern / implementation for this problem (perhaps not even a Map)

Now as much as I like writing code, I don’t like reinventing things, and I prefer reusing open source code.

I don’t mind much about the storage implementation, SQL or NoSQL, but it needs to allow a Map to be automatically persistent, and avoid keeping it entirely in memory.

Is there such a known library / implementation? is this problem familiar? am I attacking it in the right way?

---

Update:

based on comments, I’ll look into these (older, but pretty much duplicate) questions:

• Are they any decent on-disk implementations of Java's Map?

• Disk based HashMap

and vote to close this one if they answer my question and still up to date

Update2:

1. This is not an exact duplicate, I’m looking for a database backed persistence, the other questions are wider (any disk based implementation)
2. Duplicates are not always a bad thing, please read this post by Jeff Atwood before voting to close

Answer 1

Many key-value stores provide Map interface. For example, https://github.com/jankotek/JDBM3

See also SO questions:

key-value store suggestion

Java disk-based key-value storage