Question

I’m looking for a way to store a string->int mapping. A HashMap is, of course, a most obvious solution, but as I’m memory constrained and need to store 2 million pairs, 7 characters long keys, I need something that’s memory efficient, the retrieval speed is a secondary parameter.

Currently I’m going along the line of:

List<Tuple<String, int>> list = new ArrayList<Tuple<String, int>>();
list.add(...); // load from file
Collections.sort(list);

and then for retrieval:

Collections.binarySearch(list, key); // log(n), acceptable

Should I perhaps go for a custom tree (each node a single character, each leaf with result), or is there an existing collection that fits this nicely? The strings are practically sequential (UK postcodes, they don’t differ much), so I’m expecting nice memory savings here.

Answer 1

Edit: I just saw you mentioned the String were UK postcodes so I’m fairly confident you couldn’t get very wrong by using a Trove TLongIntHashMap (btw Trove is a small library and it’s very easy to use).

Edit 2: Lots of people seem to find this answer interesting so I’m adding some information to it.

The goal here is to use a map containing keys/values in a memory-efficient way so we’ll start by looking for memory-efficient collections.

The following SO question is related (but far from identical to this one).

What is the most efficient Java Collections library?

Jon Skeet mentions that Trove is “just a library of collections from primitive types” [sic] and, that, indeed, it doesn’t add much functionality. We can also see a few benchmarks (by the.duckman) about memory and speed of Trove compared to the default Collections. Here’s a snippet:

                      100000 put operations      100000 contains operations 
java collections             1938 ms                        203 ms
trove                         234 ms                        125 ms
pcj                           516 ms                         94 ms

And there’s also an example showing how much memory can be saved by using Trove instead of a regular Java HashMap:

java collections        oscillates between 6644536 and 7168840 bytes
trove                                      1853296 bytes
pcj                                        1866112 bytes

So even though benchmarks always need to be taken with a grain of salt, it’s pretty obvious that Trove will save not only memory but will always be much faster.

So our goal now becomes to use Trove (seen that by putting millions and millions of entries in a regular HashMap, your app begins to feel unresponsive).

You mentioned 2 million pairs, 7 characters long keys and a String/int mapping.

2 million is really not that much but you’ll still feel the “Object” overhead and the constant (un)boxing of primitives to Integer in a regular HashMap{String,Integer} which is why Trove makes a lot of sense here.

However, I’d point out that if you have control over the “7 characters”, you could go even further: if you’re using say only ASCII or ISO-8859-1 characters, your 7 characters would fit in a long (*). In that case you can dodge altogether objects creation and represent your 7 characters on a long. You’d then use a Trove TLongIntHashMap and bypass the “Java Object” overhead altogether.

You stated specifically that your keys were 7 characters long and then commented they were UK postcodes: I’d map each postcode to a long and save a tremendous amount of memory by fitting millions of keys/values pair into memory using Trove.

The advantage of Trove is basically that it is not doing constant boxing/unboxing of Objects/primitives: Trove works, in many cases, directly with primitives and primitives only.

(*) say you only have at most 256 codepoints/characters used, then it fits on 7*8 == 56 bits, which is small enough to fit in a long.

Sample method for encoding the String keys into long‘s (assuming ASCII characters, one byte per character for simplification – 7 bits would be enough):

long encode(final String key) {
    final int length = key.length();
    if (length > 8) {
        throw new IndexOutOfBoundsException(
                "key is longer than 8 characters");
    }
    long result = 0;
    for (int i = 0; i < length; i++) {
        result += ((long) ((byte) key.charAt(i))) << i * 8;
    }
    return result;
}