Question

For one of my data analysis pipelines, I end up generating a lot of individual CSV files. I would like to transpose them, concatenate them, and transpose them again. However, the amount of data is large, so loading it all into memory is not practical.

Answer 1

Use a generator, e.g.

from itertools import izip

file1 = open("test", "r")
file2 = open("test2", "r")

def lazy(file):
    for line in file:
        #do something with the line
        yield line

for lines in izip(lazy(file1), lazy(file2)):
    print lines

http://wiki.python.org/moin/Generators

Edit: You can use the CSV module to parse it, also I realized that the readlines() method of file objects isn’t lazy, so you have to use the for line in file pattern.