Question

I’m encountering an issue where float numbers are being truncated in my csv writing process. This is difficult to replicate, as it happens infrequently across thousands of files, but I need a protection against it. Here is an example of what the code looks like:

import csv
import numpy as np
x = np.random.normal(0, .001, 1000).tolist()
draws_header = ['draw%s'%(x) for x in range(1000)]
final_output = np.array(x)
outfile = open('filepath.csv', 'w')
writer = csv.writer('filepath')
writer.writerow(first_row)
writer.writerows(final_output)
outfile.close()

Based on the output (in which all numbers are necessarily below 1), it looks like the final characters in a small number (ie, “…e-5”) are getting lost:

draw373         draw374         draw375          draw376    
0.000744        0.003008        0.001566         9.727522

Any suggestions on how to prevent this?

Answer 1

I would suggest using numpy’s csv writer for this. For example:

>>> import numpy as np
>>> x = np.random.normal(0, .001, 1000)
>>> draws_header = ['draw%s'%(i) for i in range(1000)]
>>> f = open('file.csv', 'w')
>>> np.savetxt(f, np.array(draws_header)[:,None].T, fmt="%s", delimiter="\t")
>>> np.savetxt(f, x[:,None].T, delimiter="\t")
>>> f.close()

This serializes the numbers correctly. You can also pass a format string to savetxt to specify how to print your floating point values.