Question

I need a fastest solution for transferring data from XML file to MySQL tables.
I have XML file with multiple tables inside, like this:

<?xml version="1.0" standalone="yes"?>
<RawData xmlns="">
  <Table1>
    <ID_Table1>1</ID_Table1>
    <Name>Victor</Name>
  </Table1>
  <Table2>
    <ID_Table2>1</ID_Table2>
    <Quantity>10</Quantity>
  </Table2>
</RawData>

and within VS2010, i have DataSource with DataTables and TableAdapters from MySql db for Table1 and Table2.
My goal was to read XML file and pass its data directly to these DataTables with this:

myDSDataSet eDS = (myDSDataSet)this.FindResource("myDS"); // Declared in XAML

OpenFileDialog dlg = new OpenFileDialog();
dlg.Filter = "XML Files|*.xml";
dlg.Title = "Select a XML File";

Nullable<bool> result = dlg.ShowDialog();

if (result == true)
{                  
 mt1TableAdapter mt1_TA = new mt1TableAdapter();
 mt2TableAdapter mt2_TA = new mt2TableAdapter();
 manager = new TableAdapterManager();       

 xmlDS = new DataSet();
 dt = null;

 try
 {                     
   xmlDS.ReadXml(dlg.FileName,XmlReadMode.InferTypedSchema);

   for (int i = 0; i < xmlDS.Tables.Count; i++)
   {                          
     dt = xmlDS.Tables[i].Copy();

     eDS.Tables[eDS.Tables.IndexOf(xmlDS.Tables[i].TableName)].Merge(dt);                           
   }

   mt1_TA.Update(eDS.mt1);
   mt2_TA.Update(eDS.mt2);

   MessageBox.Show("Loading complete.");
 }
 catch (Exception error)
 {
    MessageBox.Show("ERROR: " + error.Message);
 }              
}

But after executing this code i got 2 big problems:
1. If data types differ i get exception (in DataSource field is DateTime, XML field is read as string)
2. Calling TableAdapter.Update() takes a long time to save data to db (15k rows takes 10-15mins)

So… my question is, Can someone please help me solve these two problems or give me some direction as to what is the fastest and best method for saving XML data to mysql.

Note:
– I’m using VS2010 and MySQL 5.1.
– XML file needs to be loaded from external source.
– XML has its xmlns but its omitted here for simplicity.
– I tried upgrading MySQL to 5.6 and using LOAD XML but i cant use this command within stored procedure.

Thanks

Answer 1

Ok, I’ve resolved this problem. Posting here just in case someone else comes across this problem. Following solution uses predefined schema’s for table structure and BackGroundWorker for data processing.
Idea was to input newly read data into MySQL tables using StringBuilder (fastest way to generate huge strings). StringBuilder variable takes all sql expressions and executes in single “go” on DB (reduces time of processing greatly!).

StringBuilder sqlCommand = new StringBuilder();

DataSet xmlDS = new DataSet();

try
{
   xmlDS.ReadXmlSchema(shemaPath); 
   xmlDS.ReadXml(dlg.FileName, XmlReadMode.ReadSchema);           

   sqlCommand.Clear();
   sqlCommand.Append("START TRANSACTION;");  

    for (int i = xmlDS.Tables.Count-1; i >= 0; i--)
    {
       if (xmlDS.Tables[i].Rows.Count > 0) 
          sqlCommand.Append("DELETE FROM " + xmlDS.Tables[i].TableName + "; ");
    }   

    int brojacDataTable = 0;             

    foreach (DataTable dataTable in xmlDS.Tables) 
    {
        brojacDataTable++;
        if (dataTable.Rows.Count > 0)
        {    
          sqlCommand.Append(" INSERT INTO " + dataTable.TableName + " VALUES");

          int brojacDataRows = 0;
          foreach (DataRow dataRow in dataTable.Rows)
          {
             brojacDataRows++;
             sqlCommand.Append("(");

             for (int i = 0; i < dataRow.ItemArray.Length; i++)
             {
                if (!System.DBNull.Value.Equals(dataRow.ItemArray[i]))
                {
                  if (dataRow.ItemArray[i] is System.DateTime) sqlCommand.Append("'" +((DateTime)dataRow.ItemArray[i]).ToString("yyyy-MM-dd") + "'");
                  else sqlCommand.Append("'" + dataRow.ItemArray[i].ToString() + "'");
                }
                else sqlCommand.Append("null");

               if (i < dataRow.ItemArray.Length - 1) sqlCommand.Append(",");
             }

            if (brojacDataRows < dataTable.Rows.Count) sqlCommand.Append("),");
            else sqlCommand.Append(");");
           }
        }
     }

  sqlCommand.Append("COMMIT;");

In the end, after some testing, total time for uploading roughly 30 tables with 200.000+ records was less than 20seconds 😀