Question

I’m working on an import from a CSV file to my ASP.NET MVC3/C#/Entity Framework Application.

Currently this is my code, but I’m looking to optimise:

var excel = new ExcelQueryFactory(file);
var data = from c in excel.Worksheet(0)
            select c;
var dataList = data.ToList();

List<FullImportExcel> importList = new List<FullImportExcel>();
foreach (var s in dataList.ToArray())
{
    if ((s[0].ToString().Trim().Length < 6) && (s[1].ToString().Trim().Length < 7))
    {
        FullImportExcel item = new FullImportExcel();
        item.Carrier = s[0].ToString().Trim();
        item.FlightNo = s[1].ToString().Trim();
        item.CodeFlag = s[2].ToString().Trim();

        //etc etc (50 more columns here)

        importList.Add(item);
    }
}

PlannerEntities context = null;
context = new PlannerEntities();
context.Configuration.AutoDetectChangesEnabled = false;
int count = 0;

foreach (var item in importList)
{
    ++count;
    context = AddToFullImportContext(context, item, count, 100, true);
}

private PlannerEntities AddToFullImportContext(PlannerEntities context, FullImportExcel entity, int count, int commitCount, bool recreateContext)
{
      context.Set<FullImportExcel>().Add(entity);

      if (count % commitCount == 0)
      {
          context.SaveChanges();
          if (recreateContext)
          {
              context.Dispose();
              context = new PlannerEntities();
              context.Configuration.AutoDetectChangesEnabled = false;
          }
      }
      return context;
}

This works fine, but isn’t as quick as it could be, and the import that I’m going to need to do will be a minimum of 2 million lines every month. Are there any better methods out there for bulk imports?
Am I better avoiding EF altogether and using SQLConnection and inserting that way?

Thanks

Answer 1

I do like how you’re only committing records every X number of records (100 in your case.)

I’ve recently written a system that once a month, needed to update the status of upwards of 50,000 records in one go – this is updating each record and inserting an audit record for each updated record.

Originally I wrote this with the entity framework, and it took 5-6 minutes to do this part of the task. SQL Profiler showed me it was doing 100,000 SQL queries – one UPDATE and one INSERT per record (as expected I guess.)

I changed this to a stored procedure which takes a comma-separated list of record IDs, the status and user ID as parameters, which does a mass-update followed by a mass-insert. This now takes 5 seconds.

In your case, for this number of records, I’d recommend creating a BULK IMPORT file and passing that over to SQL to import.

http://msdn.microsoft.com/en-us/library/ms188365.aspx