Question

I have some financial data that I am processing in C++. I am storing it in a simple binary format because it requires fewer resources and is fast, however I would like to add compression to the file. I am guessing I will be IO bound so the compression won’t cost me much in terms of processing speed.

I have no idea how to do the compression, as I am an academic and not a real programmer. I could really use some hand holding on this one.

I have my data in a structure like this:

  struct TradesBin {
    int ttim;
    int prc;
    int siz;
    short int g127;
    short int corr;
    char cond[2];
    char ex[1];
}__attribute__((packed));

Which I can write to a binary file as follows:

ofstream fout(outfile.c_str(), ios::out | ios::binary);
fout.write((char *) &tbin, sizeof(TradesBin));

Where tbin is filled with TradesBin data.

How do I now add compression to these files? I have heard only vaguely of things like ZLO, Bzip2, zlib, and Boost.IOStreams. I much appreciate your guidance and suggestions!

Thank you!

Answer 1

Zlib allows you to do this, but it is provides only a plain C interface. In a nutshell, you do

gzFile fp = gzopen(fname.c_str(),"wb");
gzwrite(fp, (void*) (&vec[0]), sizeof(TradesBin)*nelem);
gzclose(fp);

where vec would be for example a std::vector<TradesBin> for your TradesBin structure.