Question

I have a highly formatted file with large amount of data which I used pretty frequently so I store it into std::map and serialize it with boost::serialization library. the serialization code looks like this:

boost::iostreams::filtering_ostream ofs {};
ofs.push (boost::iostreams::zlib_compressor ());
ofs.push (boost::iostreams::file_sink (file_name));
boost::archive::binary_oarchive oa {ofs};
oa << my_map;

But every time I might just need part of the map, like a certain key-value pair. So I am wondering whether I can partially de-serialize the map from the file and only retrieving the pair I specify? In this way I don’t have to read the whole file into memory.

Answer 1

You could make an index of your data, though that’s a bit harder to use in the presence of compression. If your data file was uncompressed, and say each record was on one line, then you could make an index in which you store consecutive offsets of each line in fixed-width records, say unsigned 64-bit numbers.

To locate line 5112, you’d open the index, seek to 5112 * 8, and read two uint64_ts, say a and b. Then you open your data file, seek to a and read b - a bytes, which is your record.

If the data is compressed, you’ll have to look at your compression library and see if it allows you to partially decompress only that region of the file which contains the bytes [a, b).