I have a large number of sequential integers i need to do a lookup on, ie I need to get an offset for an serial integer id. The problem is that I’d rather not load the entire table into memory to build a hashtable/dictionary due to memory constraints so what to do?
One solution that might work is to have a file where the first integer stored is the lowest id used, then you write an array of zero integers one for each id to the largest (appending when needed) and write in the id at the correct position. For instance if the lowest id is 1000 and you want to fetch the offset at 20000 you simply retrieve the integer at position 10000+20000-1.
Using memory mapping this technique should perform pretty well. Have anyone had a similar problem, is this a good solution or is there any better way?
You can use a B-Tree, which is specifically optimized for use on hard disks.
B-Trees are used by almost all modern databases and filesystems.