Home/ Questions/Q 514089
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-13T07:29:19+00:00

I have a hash table that I want to store to disk. The list

  • 0

I have a hash table that I want to store to disk. The list looks like this:

<16-byte key                   > <1-byte result>
a7b4903def8764941bac7485d97e4f76 04
b859de04f2f2ff76496879bda875aecf 03
etc...

There are 1-5 million entries. Currently I’m just storing them in one file, 17-bytes per entry times the number of entries. That file is tens of megabytes. My goal is to store them in a way that optimizes first for space on the disk and then for lookup time. Insertion time is unimportant.

What is the best way to do this? I’d like the file to be as small as possible. Multiple files would be okay, too. Patricia trie? Radix trie?

Whatever good suggestions I get, I’ll be implementing and testing. I’ll post the results here for all to see.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You could just sort entries by key and do a binary search.

    Fixed size keys and data entries means you can very quickly jump from row to row, and storing only the key and data means you’re not wasting any space on meta data.

    I don’t think you’ll do any better on disk space, and lookup times are O(log(n)). Insertion times are crazy long, but you said that didn’t matter.

    If you’re really willing to tolerate long access times, do sort the table but then chunk it into blocks of some size and compress them. Store the offset* and start/end keys of each block in a section of the file at the start. Using this scheme, you can find the block containing the key you need in linear time and then perform a binary search within the decompressed block. Choose the block sized based on how much of the file you’re willing to loading into memory at once.

    Using an off the shelf compression scheme (like GZIP) you can tune the compression ratio as needed; larger files will presumably have quicker lookup times.

    I have doubts that the space savings will be all that great, as your structure seems to be mostly hashes. If they are actually hashes, they’re random and won’t compress terribly well. Sorting will help increase the compression ratio, but not by a ton.

    *Use the header to lookup the offset of a block to decompress and use.

    • 0