Question

I am working on a script that is going to back-fill Call Detail Records from our asterisk system into our MySQL logs database. In the code below, I am trying to ignore the duplicate keys and continue onto the next row, however when this code executes all I ever see is the first row dupe warning then the script exits (example 2, below.) Is there a an obvious error that’s causing it to fail?

Forgive my poor python etiquette, I’m very new to the language. My assumption is that even though pass might break out of an exception, the foreach loop doesn’t survive.

EDIT/NOTE: This is also worth mentioning after I solved the problem of the exiting for loop: because I had a finally: block which closed the sql connection, the finally: block was being executed after the except: pass and closing the connection. So in the above example, the finally: was still causing the program to abend.

#!/usr/bin/python -d

import csv
import sys
import MySQLdb as mdb
log="Master.csv"

try:
        con = mdb.connect('1.2.3.4','abcd','efgh','ijkl')
        cur = con.cursor()

        #Inefficient way of getting row count.
        rcount = csv.reader(open(log, 'rb'))
        print "Number of rows in csv: %d" % (len(list(rcount)))

        #OK, real csv processing now.
        reader = csv.reader(open(log, 'rb'))

        iter = 0
        for row in reader:
                print "Row: %d" % (++iter)

                clid = row[0]
                src = row[1]
                dst = row[2]
                dcontext = row[3]
                channel = row[4]
                dstchannel = row[5]
                lastapp = row[6]
                lastdata = row[7]
                start = row[8]
                end = row[10]
                duration = row[11]
                billsec = row[12]
                disposition = row[13]
                amaflags = row[14]
                accountcode = row[15]
                uniqueid = row[16]

                insertstr= "INSERT INTO cdr_extended (duration,billsec,amaflags,start,end,clid,src,dst,dcontext,channel,dstchannel,lastapp,lastdata,disposition,accountcode,uniqueid) VALUES (%s,%s,0,'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s');" % (duration,billsec,start,end,clid,src,dst,dcontext,channel,dstchannel,lastapp,lastdata,disposition,accountcode,uniqueid)
                cur.execute(insertstr)
                con.commit()

except mdb.Error, e:
        if e.args[0] == 1062:
                print "Dupe key on uniqueid: %s"  % (uniqueid)
                pass
        else:
                print "Error %d: %s" % (e.args[0],e.args[1])
                sys.exit(1)

finally:
        if con:
                con.close()

The output:

Number of rows in csv: 2696
Row: 0
Dupe key on uniqueid: 1342632723.8

Answer 1

Wrap your try...except block only around the code that would cause the exception, which is in the for loop. I don’t use MySQLdb, but something like this should work:

            try:
                insertstr= "INSERT INTO cdr_extended (duration,billsec,amaflags,start,end,clid,src,dst,dcontext,channel,dstchannel,lastapp,lastdata,disposition,accountcode,uniqueid) VALUES (%s,%s,0,'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s');" % (duration,billsec,start,end,clid,src,dst,dcontext,channel,dstchannel,lastapp,lastdata,disposition,accountcode,uniqueid)
                cur.execute(insertstr)
                con.commit()
            except mdb.Error, e:
                if e.args[0] == 1062:
                    print "Dupe key on uniqueid: %s"  % (uniqueid)
                else:
                    print "Error %d: %s" % (e.args[0],e.args[1])
                    sys.exit(1)