Question

I have multiple csv files with date as filename (20080101.csv to 20111031.csv) in a folder. The csv files have common headers. The csv file looks like this:

20080101.csv  
X ;Y; Z  
1 ; 1 ; 3  
1 ; 2 ; 6  
1 ; 3 ; 24  
2 ; 1 ; 24  
2 ; 2 ; 24  

20080102.csv   
X ;Y; Z  
1 ; 1 ; 0.1  
1 ; 2 ; 2  
1 ; 3 ; 67  
2 ; 1 ; 24  
2 ; 2 ; 24  

20080103.csv  
X ;Y; Z  
1 ; 1 ; 3  
1 ; 3 ; 24  
2 ; 1 ; 24  
2 ; 2 ; 24  

20080104.csv   
X ;Y; Z  
1 ; 1 ; 34  
1 ; 2 ; 23  
1 ; 3 ; 67  
2 ; 1 ; 24  
2 ; 2 ; 24  

… and so on. I want to write a script that would read the rows and if in a given row we have X=1 and Y=2, the whole row is copied to a new csv file along with filename giving the following output:

X ;Y ; Z ; filename  
1  ; 2 ; 6 ; 20080101  
1  ; 2 ; 2 ; 20080102  
1  ; 2 ; NA; 20080103  
1  ; 2 ; 23; 20080104 

Any idea how this can be done and any suggestions about modules that i should look into or any examples.
Thanks for your time and help.

Cheers,
Navin

Answer 1

This should do the job:

import glob
import os

outfile = open('output.csv', 'w')
outfile.write('X ; Y ; Z ; filename\n')
for filename in glob.glob('*.csv'):
  if filename == 'output.csv': # Skip the file we're writing.
    continue
  with open(filename, 'r') as infile:
    count = 0 
    lineno = 0 
    for line in infile:
      lineno += 1
      if lineno == 1: # Skip the header line.
        continue
      fields = line.split(';')
      x = int(fields[0])
      y = int(fields[1])
      z = float(fields[2])
      if x == 1 and y == 2:
        outfile.write('%d ; %d ; %g ; %s\n' % (x, y, z, filename))
        count += 1
    if count == 0: # Handle the case when no lines were found.
      outfile.write('1 ; 2 ; NA ; %s\n' % filename)
outfile.close()

Note that if you can’t control or trust the file format you may want to handle exceptions thrown by the conversions to int/float.