Question

I’m working with the MySQLdb module in Python to interact with a database. I have a situation where there is a very large list (tens of thousands of elements) which I need to insert as rows into a table.

My solution right now is to generate a large INSERT statement as a string and execute it.

Is there a smarter way?

Answer 1

There is a smarter way.

The problem with bulk insertions is that by default autocommit is enabled thus causing each insert statement to be saved to stable store before the next insert can initiate.

As the manual page notes:

By default, MySQL runs with autocommit
mode enabled. This means that as soon
as you execute a statement that
updates (modifies) a table, MySQL
stores the update on disk to make it
permanent. To disable autocommit mode,
use the following statement:

SET autocommit=0; 

After disabling
autocommit mode by setting the
autocommit variable to zero, changes
to transaction-safe tables (such as
those for InnoDB, BDB, or NDBCLUSTER)
are not made permanent immediately.
You must use COMMIT to store your
changes to disk or ROLLBACK to ignore
the changes.

This is a pretty common feature of RDBMs systems which presume that database integrity is paramount. It does make bulk inserts take on the order of 1s per insert instead of 1ms. The alternative of making an overlarge insert statement tries to achieve this single commit at risk of overloading the SQL parser.