Question

I have a challenging situation; we will have programs on Mac, PC, iOS and Android receiving files in a legacy format and parsing data from those files. We cannot change how those files are created.

The files are produced by a C++ program filling a struct with numbers and Strings and then writing it out. Here’s a sanitized version.

struct MyObject {
String Kfkj(MAXKYS); 
String Oern(MAXKYS);
String Vdflj(MAXKYS, 9); 
int Muic;
int Tdfkj;
int VdfkAsdk;
int SsdjsdDsldsk; 
int Ndsoief; 
String TdflsajPdlj; 
String TdckjdfPas; 
String AdsfakjIdd;
int IdkfjdKasdkj;
int AsadkjaKadkja(MAXKYS); 
int Kasldsdkj;
bool Usadl;
String PsadkjOasdj(9); 
String PasdkjOsdkj;
};

Primitives and Strings, as you can see.

Then here is how they write it out to a file:

MyInstance MyObject;
FileName = "C:\MyFile.ab2"
ofstream fout (FileName, ios::binary);
fout.write((char*)& MyInstance, sizeof(MyInstance));

There is no option for us to translate it once and then distribute the file to other platforms; we must translate it on each and every different platform, and this is what we have to work with. I’d appreciate any information on how C++ serializes data, so we know how to parse the file.

EDIT: solution

The feedback I received from multiple answers here was VERY helpful. Using that, I did extensive analysis with hex editors and discovered:

• the elements come in the file one after another
• a “String,” in this case, starts with an int describing how many characters follow the int for that String. If the String does not exist, it will still have that int with a value of 0.
• integers, for the files and machines I saw, are two bytes, little-endian, and MOSTLY unsigned (there were a few that were signed, just to keep me on my toes)
• the boolean was two bytes, with apparently -1 (FF FF) representing “true”

So far we have not ran into issues with different padding or endianness on different devices, but those are very real concerns. The skilled notes and warnings in these answers provides us with more ammunition to try to convince the client to change to a less fragile alternative, such as XML or JSON, for transferring data online across platforms.

As for those of you asking if the developer was fired… well, let’s just say their code is very old, but after multiple conversations we’re still having trouble convincing them writing out the C++ struct and trying to read that on different platforms is not a good idea.

Answer 1

You’re going to run into many problems.

C++ doesn’t have a specific format for serializing data per se. It is highly dependent on the computer architecture/processor that you are running on.

The compiler is allowed to add padding to help alignment on systems. When we say alignment we basically are referring to an architecture/processor’s affinity for having data lie on specific byte boundaries. For example, some processors vastly prefer floating point numbers to lie at 4 or 8 byte boundaries – if they don’t the processor may work much slower or may not work at all.

So, you can’t simply know what padding your system is adding magically.

What you can do is use #pragma pack(1) / #pragma pack(0) to stop your compiler from padding your numbers.

PS: you also have to worry about endianness. What if one computer is running on big-endian and one is little endian? They will interpret bytes differently without a conversion.

Simply put, you either have to fix the application generating the files so it uses a proper serialization scheme OR you need to look at it running on a SPECIFIC computer, look at exactly how it writes the files, and write a translator for every target platform (which is just silly).

Interesting Suggestion

If you’re really stuck, write an app that monitors the folder where you write files. Have the app pick up the files (since it’s on the same PC it’ll be able to read their format without issue). Have it write the files back in XML or some other true serialization format and distribute those instead.