I have a problem where I need to assemble a Map whose eventual size is in the GBs (going on past 64GB) and I cannot assume that a user of the program will have this kind of monster machine hanging around. A nice solution would be to distribute this map across a number of machines to make a far more modest memory footprint per instance.
Does anyone know of a library/suite of tools which can perform this sharding? I do not care about replication or transactions; just spreading this memory requirement around.
I suggest that you start with hazelcast:
http://www.hazelcast.com/
It is open-source, and in my opinion it is very easy to work with, so it is the best framework for rapid prototyping.
As far as I as know, it performs faster than the commercial alternatives, so I wouldn’t worry about performance either.
(I haven’t formally benchmarked it myself)