Question

We have to implement the import procedure.
The quantities of data is about 100 megabytes for one CSV files (the files already resides on server, so no upload necessary).

Any advice on whether to implement the bulk “LOAD DATA LOCAL INFILE” or row-by-row “INSERT/REPLACE” on importing data into database.

I’ve checked that “LOAD DATA LOCAL INFILE” works mUUUch faster than a row-by-row import, BUT the row-by-row import gives the better control on reimport/update and so forth.

In some sense, could we benefit from both of the implementation, that is speed from the first one and control from the second?

Any advice or completely different way to implement the import procedures, code snippets, database tricks are warmly appreciated.

Thank you!

Answer 1

No. Either you have full control over the process and let PHP do the insert, or you have it at full speed and let the MySQL client library handle it.

What you can do is load it into an intermediate table and then perform a query that selects from that table and inserts it into the actual table. This will allow you to do bulk manipulation of the data.