Question

I have a piece of code which generates quite large lists in each iteration. To save memory I want to write each list to a binary file in each iteration after the list has been generated.
I have tried this with text files(even setting the parameter to “wb” in linux). “wb” seems not to have any effect for the file to be written in binary or text format. Moreover, the written file is huge and I don’t want this. I am sure that If i can write these lists in binary format this file will be much smaller.
thanks

Answer 1

Since you mentioned the need for compressibility, I’d suggest using pickle with the gzip module to compress your output. You can write and read back your lists one at a time, here’s an example of how:

import gzip, pickle

output = gzip.open('pickled.gz', 'wb', compresslevel=9)

for x in range(10):
     output.write(pickle.dumps(range(10)) + '\n\n')
output.close()

And then use a generator to yield the lists back one at a time:

def unpickler(input):
    partial = []
    for line in input:
        partial.append(line)
        if line == '\n':
            obj = ''.join(partial)
            partial = []
            yield pickle.loads(obj)

input = gzip.open('pickled.gz', 'rb')
for l in unpickler(input):
    print l

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]