Question

I’m doing calculations and the resultant text file right now has 288012413 lines, with 4 columns. Sample column:

288012413; 4855 18668 5.5677643628300215

the file is nearly 12 GB’s.

That’s just unreasonable. It’s plain text. Is there a more efficient way? I only need about 3 decimal places, but would a limiter save much room?

Answer 1

Well,

• The first column looks suspiciously like a line number – if this is the case then you can probably just get rid of it saving around 11 characters per line.
• If you only need about 3 decimal places then you can round / truncate the last column, potentially saving another 12 characters per line.

I.e. you can get rid of 23 characters per line. That line is 40 characters long, so you can approximatley halve your file size.

If you do round the last column then you should be aware of the effect that rounding errors may have on your calculations – if the end result needs to be accurate to 3 dp then you might want to keep a couple of extra digits of precision depending on the type of calculation.

You might also want to look into compressing the file if it is just used to storing the results.