Question

I have a web application that loads some 50 tables from SQL Server into a DataSet, builds a more complex data structure based on it, and then caches the data so that the web application can execute faster.

The initial load of the application takes around 10 seconds, half of which is simply loading data from the database tables. Although this is reasonable in production, it gets frustrating in development. I want to speed things up.

What I have currently is similar to the following pseudo-code:

var ds = new DataSet();
var tablesToFill = new List<string>() { ... };
connectToDatabase();
foreach (var t in tablesToFill) fill(ds, t);
disconnectFromDatabase();

The fill(DataSet, string) method basically fills in something similar to an ‘Select * from {tablename};’ into the DataSet.Tables[tablename]

I wondered if loading in parallel would make things faster.

First, I added MultipleActiveResultSets=true to my connection string (SqlClient). Then, I tried the following pseudo-code:

var ds = new DataSet();
var tablesToFill = new List<string>() { ... };
connectToDatabase();
ds.EnforceConstraints=false; // without this, get concurrency errors in DataSet
tablesToFill.AsParallel().ForAll(t => fill(ds,t));
ds.EnforceConstraints=true;
disconnectFromDatabase();

This appears to work fine. On a subset of two dozen tables, load times are slashed by 66% (2.7 seconds to 0.9 seconds).

Is there anything I should be aware of that could come back and haunt me later? According to MSDN, I’m supposed to synchronize write operations on DataSets but since they’re all writing to different tables, things appear to work fine. However, am I just being lucky and/or are there scenarios (or versions of .NET) where this could bring trouble?

Thanks!

Edit: Thinking further, if you feel this is dangerous due to thread safety, how about having each fill work on its own distinct DataSet in parallel and then moving all the DataTables into a common DataSet (in the same thread). I assume I can detach & re-attach a DataTable from one DataSet to another very quickly (O(1) time, without having to duplicate any data).

Answer 1

how about having each fill work on its own distinct DataSet in parallel and then moving all the DataTables into a common DataSet (in the same thread)

Why not fill separate DataTables and then add the DataTables to the DataSet.

Wait until the last DataTable is filled before putting them in the DataSet and there should be no issues.