Question

I have a batch of ‘robots’ that runs around reading RSS threads and stores the results in a database, and I parallelized it so that many feeds can be fetched at once:

Parallel.ForEach(context.Feeds, feed => ProcessRssFeed(feed, context));
context.SubmitChanges();

The ‘ProcessRssFeed’ function can insert records into the context as they are found, and each feed could be anywhere from zero to hundreds of items. There are a lot of feeds, so I did not want to create a LINQ DataContext for each.

I’m worried, though, that I could be accumulating thousands upon thousands of records on the client. I suppose I could run out of memory.

Since there’s no concurrency issue here if it were possible I’d like to tell the DataContext “go ahead and submit records periodically if you like”. Is there some practical way to achieve this?

Answer 1

I would recommend creating a new DataContext for each. The DataContexts are quite light weight compared to the actual database connection. The DataContext uses connection pooling when connecting to the database, you don’t get much overhead by using separate DataContexts.

Keep only the stuff that need to be submitted atomically in a DataContext, submit that and create a new DataContext for the next item.

There is no built in method for periodically submit, but you can watch the number of items in DataContext.GetChangeSet() and submit when that count is over a given threshold. But you should only do that if profiling shows that creating new DataContexts is really a bottleneck in your system.