Question

I have a big csv file with datetime and value recorded every 10 seconds. The csv file looks like this:

Datetime             Data  
2008-10-01 12:00:10, 34  
2008-10-01 12:00:20, 55  
2008-10-01 12:00:30, 46  
2008-10-01 12:00:40, 33  
2008-10-01 12:00:50, 55  
2008-10-01 12:01:00, 21  
2008-10-01 12:01:10, 2  
2008-10-01 12:01:20, 34  
2008-10-01 12:01:30, 521  
2008-10-01 12:01:40, 45  
2008-10-01 12:01:50, 32  
2008-10-01 12:02:00, 34  

I want to write a script that would calculate minute average and write it in a new csv file giving the following output:

Datetime             Data  
2008-10-01 12:00:00, 40.67  
2008-10-01 12:01:00, 111.33  

Any idea how this can be done and any suggestions about modules that I should look into or any examples.

Answer 1

Use the csv.reader to parse the file and a dictionary to cluster the results. The str.rpartition method can split-off the seconds. Use sum and len to compute the average:

data = '''\
2008-10-01 12:00:10, 34  
2008-10-01 12:00:20, 55  
2008-10-01 12:00:30, 46  
2008-10-01 12:00:40, 33  
2008-10-01 12:00:50, 55  
2008-10-01 12:01:00, 21  
2008-10-01 12:01:10, 2  
2008-10-01 12:01:20, 34  
2008-10-01 12:01:30, 521  
2008-10-01 12:01:40, 45  
2008-10-01 12:01:50, 32  
2008-10-01 12:02:00, 34
'''.splitlines()

import csv

d = {}
for timestamp, value in csv.reader(data):
    minute, colon, second = timestamp.rpartition(':')
    if minute not in d:
        d[minute] = [float(value)]
    else:
        d[minute].append(float(value))

for minute, values in sorted(d.items()):
    avg_value = sum(values) / len(values)
    print minute + ',' + str(avg_value)