Question

I’m loading a csv file into my database via a web form.

The order of the raw data is consistent in each csv file, but it changes from file to file, depending on the source, so I have a preview form that shows five rows and allows you to assign a column via a drop-down list of valid column names in the table.

Then I use the cgi form to build an INSERT statement, and parse the csv file line-by-line to populate the table.

But it is running EXTREMELY slow. I’m concurrently populating two tables, one with 961402 rows (7 columns with values), and the other with 1835538 rows(1 column with values), and each has been running for at least half an hour. I’m only seeing something like 100 new rows per second.

Can you see anything here that would slow me down?

NOTE: I know there is some ugly code in here, it was one of the first python cgi scripts I wrote while figuring this language out.

 for item in form:
          field = form.getvalue(item)
          field = cgi.escape(field)
          if field == 'null':
                  pass
          elif item == 'csvfile':
                  pass
          elif item == 'campaign':
                  pass
          elif item == 'numfields':
                  pass
          else:
                  colname = str(colname) + ", " + str(item)

                  colnum.append(field)
  assert(numfields > 0)
  placeholders = (numfields-1) * "%s, " + "%s"
  query = ("insert into %s (%s)" % (table, colname.lstrip(",")))
  with open(fname, 'rb') as f:
          reader = csv.reader(f)
          try:
                  record = 0
                  errors = 0
                  for row in reader:
                          try:
                                  record = record + 1
                                  data = ''
                                  for value in colnum:
                                          col = int(value)
                                          rawrow = row[col]
                                          saferow = rawrow.replace("'", "-")
                                          saferow = saferow.replace("-", "")
                                          data = str(data) + ", '" + saferow + "'"
                                  dataset = data.lstrip(',')
                                  insert = query + (" values (%s)" % dataset)
                                  cur.execute(insert)
                                  con.commit()
                                  print ".",
                          except IndexError, e:
                                  print "Row:%d file %s, %s<br>" % (reader.line_num, fname.lstrip("./files/"), e)
                                  errors = errors + 1
                          except csv.Error, e:
                                  print "Row:%s file %s, line %d: %s<br>" % (record, fname, reader.line_num, e)
                                  errors = errors + 1
                          except mdb.Error, e:
                                  print "Row:%s Error %d: %s<br>" % (record, e.args[0], e.args[1])
                                  errors = errors + 1
                          except:
                                  t,v,tb = sys.exc_info()
                                  print "Row:%s %s<br>" % (record, v)
                                  errors = errors + 1
          except csv.Error, e:
                  print "except executed<br>"
                  sys.exit('file %s, line %d: %s' % (fname, reader.line_num, e))
  print "Succesfully loaded %s into Campaign %s, <br>" % (fname.lstrip("./files/"), table)
  print record - errors, "new records.<br>"
  print errors, "errors.<br>"

EDIT/UPDATE: Using LOAD DATA LOCAL INFILE worked like a charm, I loaded up 600K records in less than a minute.

New Code is cleaner, too.

    else:
            colnum.append([field, item])
sortlist =  sorted(colnum, key=itemgetter(0))
cols = ''
for colname in sortlist:
    cols = cols + "%s, " % colname[1]
cur.execute("LOAD DATA LOCAL INFILE '%s' IGNORE INTO TABLE %s FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (%s)" % (fname, table, cols.rstrip(', ')))
con.commit()

The only catch is that I have to do a smidge more work preparing my csv files to ensure data integrity, otherwise, works like a charm.

Answer 1

INSERT INTO, done one row at a time, is pretty slow considering that some SQLs, like mySQL, support either having a bunch of rows on a single insert command or LOAD DATA statements that can read CSV files quickly into the server.

See also: https://dba.stackexchange.com/questions/16809/why-is-load-data-infile-faster-than-normal-insert-statements