Question

I know I am probably not doing this the most efficient way. I have data stored in XML format, and I am parsing the XML data into individual INSERT INTO .. VALUES (..) statements and then executing these statements. Is there a good way of executing millions of these INSERT INTO statements as a batch?

I cannot use regular SQL server utility tools to import the XML data because I am running this on AWS.

Any suggestions are appreciated!

Answer 1

Using the multi INSERT method, where you supply many sets of VALUES is always better than doing them individually. You should construct statements as large as your max_allowed_packet setting is, and ideally adjust that to be as high as possible. 2GB is generally as high as it can go, and for maximum speed you’ll be wanting to sling in at least 10-50MB sized chunks.

LOAD DATA INFILE is also an option, but this is usually at most 10-15% faster than the multi-INSERT method and requires the data to be in a particular format to start. CSV or tab delimited are the easiest to work with, XML is not an option.

For maximum performance, disable any indexes prior to inserting a lot of data. You can see some of these tricks even in a mysqldump snapshot of your empty database schema.

To disable indexing, execute this statement before your import:

 ALTER TABLE `table_name` DISABLE KEYS;

Then after the import, to rebuild them:

 ALTER TABLE `table_name` ENABLE KEYS;