Home/ Questions/Q 880941
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T12:10:50+00:00

So I’m trying to import some sales data into my MySQL database. The data

  • 0

So I’m trying to import some sales data into my MySQL database. The data is originally in the form of a raw CSV file, which my PHP application needs to first process, then save the processed sales data to the database.

Initially I was doing individual INSERT queries, which I realized was incredibly inefficient (~6000 queries taking almost 2 minutes). I then generated a single large query and INSERTed the data all at once. That gave us a 3400% increase in efficiency, and reduced the query time to just over 3 seconds.

But as I understand it, LOAD DATA INFILE is supposed to be even quicker than any sort of INSERT query. So now I’m thinking about writing the processed data to a text file and using LOAD DATA INFILE to import it into the database. Is this the optimal way to insert large amounts of data to a database? Or am I going about this entirely the wrong way?

I know a few thousand rows of mostly numeric data isn’t a lot in the grand scheme of things, but I’m trying to make this intranet application as quick/responsive as possible. And I also want to make sure that this process scales up in case we decide to license the program to other companies.

UPDATE:

So I did go ahead and test LOAD DATA INFILE out as suggested, thinking it might give me only marginal speed increases (since I was now writing the same data to disk twice), but I was surprised when it cut the query time from over 3300ms down to ~240ms. The page still takes about ~1500ms to execute total, but it’s still noticeably better than before.

From here I guess I’ll check to see if I have any superfluous indexes in the database, and, since all but two of my tables are InnoDB, I will look into optimizing the InnoDB buffer pool to optimize the overall performance.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    LOAD DATA INFILE is very fast, and is the right way to import text files into MySQL. It is one of the recommended methods for speeding up the insertion of data -up to 20 times faster, according to this:

    https://dev.mysql.com/doc/refman/8.0/en/insert-optimization.html

    Assuming that writing the processed data back to a text file is faster than inserting it into the database, then this is a good way to go.

    • 0