Question

I have a binary file with the following structure:

+———+——-+——+—-+——-+
| Header | Obj1 | Obj2 | … | Index |
+———+——-+——+—-+——-+

The Index is a vector of variable size depending on the number of objects and stores the file position for each object. E.g.

vector<size_t> index;

index.push_back(ofs.tellp());
write(reinterpret_cast<char *> obj, sizeof(obj));

Also the files are pretty big (1Gb+).

I’d like to put the index just after the header, so it’s faster to read. Can I do this without having to copy and write all the objects twice? Thanks.

Answer 1

You have two solutions:

• use a separate file for the index
• move on to a chunk based design

The first one is relatively obvious, so I’ll only outline the second one: the problem here is that your index size depends on the number of objects you stream. It need not be so.

A solution would be to skip a definite amount of space (later used for indexing), stream the objects (up to N), record where you are, get back to the index to write it down, and then move on to the next chunk (note: the chunks are not fixed size here).

Example of layout:

- Step 1: skip index space and start streaming objects

| Header | <unused space for now> | Obj 1 | Obj 2   | O3 | ... Obj N |


- Step 2: after writing N objects write the index,
          record the offset,
          start a new chunk

| Header | I1 | ... | IN | Offset | Obj 1 | Obj 2   | O3 | ... Obj N | <unused sp

Your index is thus built as a linked list of fixed size chunks (N) interleaved in the middle of your storage.

Note: a third solution would be to use a simple SQLite file and let it index for you…