Question

I have something like:

import java.util.HashMap;
import java.util.List;

public class A {
    HashMap<Long, List<B>> hashMap = new HashMap<Long, List<B>>();
}

class B{
    int a;
    int b;
    int c;
}

And I want to store this in database, because it will be huge huge.

I will have more 250000000 keys in HashMap and each key representing huge list of data (say list size may be around 1000).

How I can do this for best performance on retrieving list of B’s objects with Long hashKey from database?

Any other suggestions?

Thanks in advance.

Answer 1

As you have a very large data set of up to 1/4 bn * 20 * 1 k or about 5 TB, the main problem you have is that it can’t be stored in memory and is too large to store on SSD, so you will have to access disk efficiently otherwise you are looking at a latency of about 8 ms per key This should be your primary concern otherwise it will take days just to access every key randomly once.

Unless you have a good understand of how to implement this with memory mapped files you will need to use a database, preferable one design to handle large numbers of records. You with also need a disk sub-system not only for capacity but to give you more spindles so you can increase the number of requests you can perform concurrently.