Question

I’m in the process of writing a script that chunks a large CSV file down into smaller chunked files. It cross references a log file that holds the last timestamp that was chunked so that only the timestamps that are later than the logged time are written/chunked.

The first column of the csv file has a timestamp that is in %Y%m%d %H%M%S form. The CSV file also has four rows of header information that I don’t want/need in my script, which the rows in ts_pre clause removes.

The log_lookup() function just pulls the last timeseries from the log for a particular station’s CSV file that I’m viewing. Obviously, I’m working with six different stations that all have different columns of information, except they all share the same structure as I described in the second paragraph.

The partial script is:

import csv, sys, datetime

def log_lookup():  
    global STN_num
    global STN_date
    with open('/home/log.txt', 'rb') as open_log:    
        log_file = csv.reader(open_log)
            for row in log_file:
                for item in row:
                     STN_date.append(item)
            if find == 'STN_1':
                return STN_date[1]
            if find == 'STN_2':           
                return STN_date[2]
            if find == 'STN_3':
                return STN_date[3]
            if find == 'STN_4':
                return STN_date[4]
            if find == 'STN_5':
                return STN_date[5]
            if find == 'STN_6':
                return STN_date[6]

def get_ts(line):
    print line[0:19]
    return datetime.datetime.strptime(line, "%Y/%m/%d %H:%M:%S")

def main():
    log = str(log_lookup()) #useful for knowing when to start chunking 
    log_datetime = datetime.datetime.strptime(log, "%Y/%m/%d %H:%M:%S")       

    with open(sys.argv[1], 'rb') as open_file: 
        ts_from_file = csv.reader(open_file)
        for genrows in ts_from_file:            
            ts_pre.append(genrows)  
            for rows in ts_pre:             
                if rownum < 4:
                    ts_pre.pop()
                    rownum += 1
                else:       
                    for line in rows:                            
                        if get_ts(line) > log_datetime:                 
                            timeseries.append(line)

The log file is simply:

0
2011/10/06 18:40:00
2012/06/27 13:25:00
1900/01/01 00:00:00
2011/08/03 14:55:00
2012/06/27 20:05:00
2011/10/03 19:25:00

… with 0 as a placeholder. (Is it obvious I’m not a programmer?)

An example CSV file looks like:

"2011/10/03 16:40:00",0,0
"2011/10/03 16:45:00",1,0
"2011/10/03 16:50:00",2,0
"2011/10/03 16:55:00",3,0

The error I’m getting when the ts_line(line) function is that it’s saying line[0:19] is:

2011/10/03 16:40:00
0

and the function returns 0 and Python throws this error:

ValueError: time data '0' does not match format '%Y/%m/%d %H:%M:%S'

I have verified that the 0 that’s returned is the second item in the CSV file, but I’m confused as to why Python returns it at all in my slice selection. Can someone explain to me why it’s returning that value and what I need to do to get the timestamp to compare to the log timestamp?

For extra credit, any advice on coding/style is always appreciated and/or advice for better ways accomplish what I’m doing. The CSV files I’m looking at are quite large (~8 MB) so the more efficient the better.

Answer 1

It looks like you are calling get_ts for each column in every line of the csv.

Instead of

for line in rows:                            
    if get_ts(line) > log_datetime:                 
        timeseries.append(line)

Try:

if get_ts(line[0]) > log_datetime:                 
    timeseries.append(line)