Question

I have a 1GB sql text file I’m importing into MySQL.
I’ve used a ‘large text file viewer’ and can see it a std mysql table export – starts with drop table, then create new table and then insert. Note: I also have a csv of the table.

I’m used the following methods to import it:

1. php.ini – upload_max_filesize, memory_limit and post_max_size (increase sizes of these but it still said it was too large – made these numbers very large – 10 zeros after default number).

2. C:\wamp\bin\mysql\mysql5.1.53\bin>mysqlimport -h localhost -u root -p –fields-t
   erminated-by=’\t’ –lines-terminated-by=’\n’ –local testing c:\temp\filename.csv

3. source c:\temp\filename.sql

I can see the second 2 methods ‘in task manager’ seem to be moving the entire 1GB file into memory before they try to insert them into MySQL. I can see the process for each and it slowly grows to over 1GB and failed with a error.

Question: Is there a preferable and fastest way to import a table? is there a way that doesn’t need to move the entire file into memory? First time I’ve worked with such a large sql table.

thx

Answer 1

Part of the problem of having such a huge export is (presumably) due to the redundancies in it. There are probably thousands of “INSERT” commands which when loaded all at once, takes up a huge chunk of memory.

If you have the exact same data as a CSV, what I would do is create a small PHP script to go line-by-line through the CSV file and create an INSERT sql query dynamically and then execute it. This should keep the memory footprint very low since you could keep replacing the values of your temporary variables (corresponding to the fields) after each insertion.

I’ve linked to the function that I’d try to use. It’s called fgetcsv. That link also has some sample scripts which I believe might be of use to you that were created by other users.

Good luck!

fgetcsv PHP function to use for the line-by-line read