Question

We use strongly typed DataSets in our application. When importing data we use the convenient DataSet.Merge() operation to copy DataRows from one DataSet to another.

StringCollection lines = ReadFromFile(fileName);
foreach (string line in lines)
{
    DataRow dr = ImportRow(line);
    dataSet1.Merge( new DataRow[] { dr } );
}
DoAdditionalCalculationsWith(dataset1);
SaveToDatabase(dataSet1);

Unfortunately this doesn’t scale. For larger imports the Merge takes up 80% of our total import time (according to our profiler).

Is there a faster way to do this?

Edit: We can’t just add the rows because they might already be in the DataSet and doing it in the database is also not an option, because our import logic is quite complex.

Answer 1

You’ve probably already tried this, but just in case:

DataSet.Merge takes an array or DataRows as a parameter.

Have you tried batching the merge, i.e. doing the following?

dataSet1.Merge(lines.Select(line=>ImportRow(line)).ToArray());

However, it’s quite possibly the case that you cannot improve performance – maybe you can avoid the need to Merge in the first place somehow – such as by doing the merge in the DB, as Sklivvz suggests.