Question

I need some advice on transferring a large amount of data to a MySQL database hosted remotely. (godaddy)

I have received a 700k row CSV file and imported it into MSSQL on my dev machine and wrote a few quick utilities.

The first was to create smaller 1 meg (max allowed by host) csv files that could be directly imported via
phpMyAdmin.
max of 1 meg http://xtubedownloads.com/public/starfield.png

I was only getting approximately 1600 rows for each meg which would mean about 437 individual manual uploads. Not feasible so I abandoned that technique.

Next I tried using the MySQL .NET connector and transfer directly through ODBC from MS Access and MS Excel. That failed.

Finally I wrote a .NET winform app that gets the data from my local SQL Server database then updates the remote MySQL database via standard insert queries using the .NET MySQL data assemblies.

I have the winform app spawn 4 threads, each inserting a new unique record to maximize my inputs at four new records for each loop but still only getting about 100k records each day.

(I have tried increasing to 10 threads and running multiple instances of the app but these transfers are memory hogs and 4 is about all my machine can take without problems)

I have locally installed the mysql admin tools (MySQL WorkBench 5.2CE) hoping to be able to somehow transfer the MSSQL data through the mysql admin tool. Not able to do that.

All of the Machine ODBC attempts are failing using standard tools such as MSAcess and MSExcel. (Could be some godaddy config issues or the fact that I’m running a 64 bit OS and the Mysql odbc tools are 32?)

I do have the ability to connect the godaddy hosted MySQL server remotely so my only other idea is to put the 700k records in an Access database on the server and create 10 or 20 WCF services to do the work of inserting to the MySQL database.

Perhaps removing the latency of having to transfer each row over the wire would drastically increase the insert speeds?

There has got to a method that is more efficient that what I have now, a machine running 24/7 for a number of days just to insert records.

Answer 1

What about something like this:

• Use a php script to retrieve a compressed version of SQL file (with, say, file_get_contents()) and save it in the server local to the MySQL. You should be able to download more data than you would be able by using HTML upload forms. I assume that you can’t do it via FTP, by the way 🙂

• Use Bigdump, a staggered import tool, to import the data into MySQL piece by piece. This way you’ll avoid timeout / other related issues.

Good luck!