Question

I have a large number of text files containg data arranged into a fixed number of rows and columns, the columns being separated by spaces. (like a .csv but using spaces as the delimiter). I want to extract a given column from each of these files, and write it into a new text file.

So far I have tried:

results_combined = open('ResultsCombined.txt', 'wb')

def combine_results():
    for num in range(2,10):  
        f = open("result_0."+str(num)+"_.txt", 'rb') # all the text files have similar filename styles
        lines = f.readlines()   # read in the data
        no_lines = len(lines)   # get the number of lines

             for i in range (0,no_lines):
                 column = lines[i].strip().split(" ")

                 results_combined.write(column[5] + " " + '\r\n')

             f.close()

if __name__ == "__main__":
    combine_results()

This produces a text file containing the data I want from the separate files, but as a single column. (i.e. I’ve managed to ‘stack’ the columns on top of each other, rather than have them all side by side as separate columns). I feel I’ve missed something obvious.

In another attempt, I manage to write all the separate files to a single file, but without picking out the columns that I want.

import glob

files = [open(f) for f in glob.glob("result_*.txt")]  
fout = open ("ResultsCombined.txt", 'wb')

    for row in range(0,488):
      for f in files:
          fout.write( f.readline().strip() )
          fout.write(' ')
      fout.write('\n')

 fout.close()

What I basically want is to copy column 5 from each file (it is always the same column) and write them all to a single file.

Answer 1

If you don’t know the maximum number of rows in the files and if the files can fit into memory, then the following solution would work:

import glob

files = [open(f) for f in glob.glob("*.txt")]

# Given file, Read the 6th column in each line
def readcol5(f):
    return [line.split(' ')[5] for line in f]

filecols = [ readcol5(f) for f in files ]
maxrows = len(max(filecols, key=len))

# Given array, make sure it has maxrows number of elements.
def extendmin(arr):
    diff = maxrows - len(arr)
    arr.extend([''] * diff)
    return arr

filecols = map(extendmin, filecols)

lines = zip(*filecols)
lines = map(lambda x: ','.join(x), lines)
lines = '\n'.join(lines)

fout = open('output.csv', 'wb')
fout.write(lines)
fout.close()