Question

I tried to load huge data file eg 10 mb files into the db using Load Data Local Infile into mysql using a windows service. But it failed and the windows service stopped without any exception. I tried with files of small size such as 2 mb and it succeded.

Is there any way I can load huge files into the mysql db? And what may be the reason for the windows service to stop?

here is my code..

    public int UpdateDataBase(string query, string filename, Logger swLog)
    {
        int status = 0;

        using (MySqlConnection myconnection = new MySqlConnection(connectionCdrBank))
        {

            try
            {
                myconnection.ConnectionTimeout = 1000;
                myconnection.Open();
                myconnection.BeginTransaction();

                MySqlCommand mycommand = new MySqlCommand("ClearTempTable", myconnection);
                mycommand.CommandType = CommandType.StoredProcedure;
                status = mycommand.ExecuteNonQuery();

                MySqlCommand mycommand1 = new MySqlCommand(query, myconnection);
                mycommand1.CommandTimeout = 1000;
                status = mycommand1.ExecuteNonQuery();

                MySqlCommand mycommand2 = new MySqlCommand("InsertToCdrDetailsTempTable", myconnection);
                mycommand2.CommandType = CommandType.StoredProcedure;
                mycommand2.CommandTimeout = 1000;
                status = Convert.ToInt32(mycommand2.ExecuteScalar());

                myconnection.Commit();
                myconnection.Close();

            }
            catch (Exception ex)
            {
                Console.WriteLine("Error:" + ex.Message);
                if (myconnection.State == ConnectionState.Open)
                    myconnection.Rollback();
                status = (ex.Message.IndexOf("Duplicate entry") != -1) ? -1000 : 0;
                swLog.WriteErrorToLog("");
                swLog.WriteErrorToLog("Error while saving: " + ex.Message);
                mail.SentMail("Error while saving:", ex.Message);
                swLog.CloseLogger();
            }
        }

        return status;
    }

where query is

LOAD DATA LOCAL INFILE 'a.txt' INTO TABLE tbl_cdrload" FIELDS TERMINATED BY ',' LINES     TERMINATED BY '\n';

Answer 1

I’m loading up to 100 MB files into MySQL Blob fields. And this is just my artificial limit.

So, 10MB files should not be a problem.

Did you set max_allowed_packet correctly?

What do you use to load the files? Your own code or some tool? What MySQL Connector?