I need to serialize some data in a binary format for efficiency (datalog where 10-100MB files are typical), and I’m working out the formatting details. I’m wondering if realistically I need to worry about file corruption / error correction / etc.
What are circumstances where file corruption can happen? Should I be building robustness to corruption into my binary format? Or should I wrap my nonrobust-to-corruption stream of bytes with some kind of error correcting code? (any suggestions? I’m using Java) Or should I just not worry about this?
edit: preliminary binary format, as I have it right now, contains a bunch of variable-length segments, so I am slightly worried that if I do ever have data corruption then upon reading it back, I could get out of sync, and cannot recover + I lose the rest of the file.
I went with a Reed-Solomon encoding system. There’s a fairly easy-to-use Java implementation of it in Java in the Google zxing library.