Question

I have to write numerical data to binary files. Since some of the data vectors I deal with can be several gigs in size, I have learned not to use C++ iostreams. Instead I want to use C File*. I’m running into a problem right off the bat where I need to write some meta data to the front of the binary file. Since some of the meta data is not known at first I need to append the meta data as I get it to the appropriate offsets in the file.

for example lets say I have to enter a uint16_t representation for the year, month , and day, but first I need to skip the first entry(a uint32_t value for precision);

I don’t know what i’m doing wrong but I can’t seem to append the file with “ab”.
Here’s an example of what I wrote:

#include<cstdio>

uint16_t year = 2001;
uint16_t month = 8;
uint16_t day = 23;

uint16_t dateArray[]={year , month, day};

File * fileStream;
fileStream = fopen("/Users/mmmmmm/Desktop/test.bin" , "wb");

if(fileStream){

// skip the first 4 bytes
fseek ( fileStream , 4 , SEEK_SET );

fwrite(dateArray, sizeof(dateArray[0]) ,( sizeof(dateArray) / sizeof(dateArray[0]) ), filestream);

fclose(filestream);

}
 // loops and other code to prepare and gather other parameters

// now append the front of the file with the precision.

uint32_t precision = 32;

File *fileStream2;
fileStream2 = fopen("/Users/mmmmmm/Desktop/test.bin" , "ab"); 

if(fileStream2){

// Get to the top of the file
rewind(fileStream2);

fwrite(&precision, sizeof(precision) , 1 , fileStream2);

fclose(fileStream2);

}

The appended data does not write. If I change it to “wb”, then the file overwrites. I was able to get it to work with “r+b”, but I don’t understand why. I thought “ab” would be proper. Also , should I be using buffers or is this a sufficient approach?

Thanks for the advise

BTW this is on MacOSX

Answer 1

Due to the way that hard drives and filesystems work, inserting bytes in to the middle of a file is really slow and should be avoided, especially when dealing with multi-gigabyte files. If your metadata is stored in to a fixed-size header, just make sure that there’s enough space for it before you start with the other data. If the header is variably sized, then chunk up the header. Put 1k of header space at the beginning, and have 8 bytes reserved to contain the value of the offset to the next header chunk, or 0 for EOF. Then when that chunk gets filled up, just add another chunk to the end of the file and write its offset to the previous header.

As for the technical IO, use the fopen() modes of r+b, w+b, or a+b, depending on your need. They all act the same with minor differences. r+b opens the file for reading and writing, starting at the first byte. It will error if the file doesn’t exist. w+b will do the same, but create the file if it doesn’t exist. a+b is the same as r+b, but it starts with the file pointer at the end of the file.

You can navigate the file with fseek() and rewind(). rewind() moves the file pointer back to the beginning of the file. fseek() moves the file pointer to a specified location. You can read more about it here.