Question

I need to insert large amount of data into SqlServer 2008. My project is based on linq-to-sql.

I process csv file with 100.000 rows. Each row is mapped to Order object. Order contains also collection of Category and Code objects. I need to map each row to object in order to validate it.

Then I need to insert all these objects into database.

List<Order> orders = Import("test.csv");
db.Orders.InsertAllOnSubmit(orders);
db.SubmitChanges();

OR

foreach(Order order in orders)
db.Orders.InsertOnSubmit(order);
db.SubmitChanges();

Both ways are slow. Is there any workaround? I may use other approach than l2sql for this task.

I read about SqlBulkCopy class – would it handle inserting child entities as well?

Answer 1

Try using smaller transactions.

foreach(List<Order> orderbatch in orders.Batch(100))
{
  db.Orders.InsertOnSubmit(orderbatch); 
  db.SubmitChanges();   
}


public static IEnumerable<List<T>> Batch<T>(this IEnumerable<T> source, int batchAmount)
{
  List<T> result = new List<T>();
  foreach(T t in source)
  {
    result.Add(t);
    if (result.Count == batchSize)
    {
      yield return result;
      result = new List<T>();
    }
  }
  if (result.Any())
  {
    yield return result;
  }
}