Question

I have a database table with a unique string field and a couple of integer fields. The string field is usually 10-100 characters long.

Once every minute or so I have the following scenario: I receive a list of 2-10 thousand tuples corresponding to the table’s record structure, e.g.

[("hello", 3, 4), ("cat", 5, 3), ...]

I need to insert all these tuples to the table (assume I verified neither of these strings appear in the database). For clarification, I’m using InnoDB, and I have an auto-incremental primary key for this table, the string is not the PK.

My code currently iterates through this list, for each tuple creates a Python module object with the appropriate values, and calls “.save()”, something like so:

@transaction.commit_on_success
def save_data_elements(input_list):
    for (s, i1, i2) in input_list:
        entry = DataElement(string=s, number1=i1, number2=i2)
        entry.save()

This code is currently one of the performance bottlenecks in my system, so I’m looking for ways to optimize it.

For example, I could generate SQL codes each containing an INSERT command for 100 tuples (“hard-coded” into the SQL) and execute it, but I don’t know if it will improve anything.

Do you have any suggestion to optimize such a process?

Thanks

Answer 1

You can write the rows to a file in the format
“field1”, “field2”, .. and then use LOAD DATA to load them

data = '\n'.join(','.join('"%s"' % field for field in row) for row in data)
f= open('data.txt', 'w')
f.write(data)
f.close()

Then execute this:

LOAD DATA INFILE 'data.txt' INTO TABLE db2.my_table;

Reference