Question

I have to copy and do some simple processing on file. I can not read whole file to the memory because it is to big. I come up with piece of code which looks like this:

buffer = inFile.read(buffer_size)
while len(buffer) > 0:
    outFile.write(buffer)
    simpleCalculations(buffer)
    buffer = inFile.read(buffer_size)

simpleCalculations procedure is irrelevant in this context but I am worried about subsequent memory allocations of buffer list. On some hardware configuration memory usage gets very high and that apparently kills the machine. I would like to reuse buffer. Is this posible in python 2.6?

Answer 1

I don’t think there’s any easy way around this. The file.read() method just returns a new string each time you call it. On the other hand, you don’t really need to worry about running out of memory — once you assign buffer to the newly-read string, the previously-read string no longer has any references to it, so its memory gets freed automatically (see here for more details).