Question

In a C# program I am importing a large text file (300mb) into a MySQL database via the MySqlBulkLoader function of the MySql .Net Connector.

The import takes quite a long time and results in almost 100% disk usage on the Windows 2003 Server it is running on. In an attempt to speed up the import the program nows splits the big file into smaller chunks.

Would it be possible to read the small file chunk (8mb) into memory (ie array) and then pass it to the MySQLBulkLoader as a file?

The bulk loader looks for a filename path:

 MySql.Data.MySqlClient.MySqlBulkLoader myBulk = new MySql.Data.MySqlClient.MySqlBulkLoader(connection);                
 myBulk.Timeout = 10 * 60; /
 myBulk.TableName = "some_table";                
 myBulk.Local = true;
 myBulk.LineTerminator = @"\n";
 myBulk.FileName = aFile.FullName;
 myBulk.FieldTerminator = "";

Answer 1

Memory isn’t a file, so the short answer is no. The alternatives are:

1. Read through the file, writing it out as a temporary file (System.IO.Path.GetTempFileName() is your friend here, for the name to give the partial file) and passing that filename to MySqlBulkLoader
2. Use a “RAM Disk” tool to create a memory based disk to put a copy of the full 300Mb file on, then pass that files path to MySqlBulkLoader.