Question

I’m parsing a big CSV file using csv.DictReader.

quotes=open( "file.csv", "rb" )
csvReader= csv.DictReader( quotes )

Then for each row I’m converting the time value in the CSV in datetime using this :

for data in csvReader:
    year = int(data["Date"].split("-")[2])
    month = strptime(data["Date"].split("-")[1],'%b').tm_mon
    day = int(data["Date"].split("-")[0])
    hour = int(data["Time"].split(":")[0])
    minute = int(data["Time"].split(":")[1])
       
    bars = datetime.datetime(year,month,day,hour,minute)

Now I would like to perform actions only on the rows of the same day. Would it be possible to do it in the same for loop or should I maybe save the data out per day and then perform actions? What would be an efficient way of baking the parsing?

Answer 1

As jogojapan has pointed out, it is important to know whether we can assume that the CSV file is sorted by date. If it is, then you could use itertools.groupby to simplify your code. For example, the for loop in this code iterates over the data one day at time:

import csv
import datetime
import itertools

with open("file.csv", "rb") as quotes:
    csvReader = csv.DictReader(quotes)

    lmb = lambda d: datetime.datetime.strptime(d["Date"], "%d-%b-%Y").date()
    for k, g in itertools.groupby(csvReader, key = lmb):
        # do stuff per day
        counts = (int(data["Count"]) for data in g)
        print "On {0} the total count was {1}".format(k, sum(counts))

I created a test “file.csv” containing the following data:

Date,Time,Count
1-Apr-2012,13:23,10
2-Apr-2012,10:57,5
2-Apr-2012,11:38,23
2-Apr-2012,15:10,1
3-Apr-2012,17:47,123
3-Apr-2012,18:21,8

and when I ran the above code I got the following results:

On 2012-04-01 the total count was 10
On 2012-04-02 the total count was 29
On 2012-04-03 the total count was 131

But remember that this will only work if the data in “file.csv” is sorted by date.