Question

We have a performance problem in production, we are trying to solve. During our attempt to solve it, we have our eyes on the datatable.

Our flow:

1. We run a select query on the database
2. Put the data into a DataTable
3. We write out all the data into a StringBuilder with a separator
4. The StringBuilder writes all text into the Response.Write, which builds a CSV file

The problem is in production, our client says the operation times out. We can’t reproduce the timeout in development, which might be because of the amounts of data in production.

Step 1:

We’ve run a profiler on the select query, and that runs really fast.

Step 2:

We have the following code:

SqlDataReader reader = searchQuery.ExecuteReader();
returnTable.Load(reader);

Step 3:

We iterate the returnTable like this:

foreach (DataRow order in orders.Rows)
{
      trackingNumber = order["TrackingNumber"] != null ? order["TrackingNumber"].ToString() : errString;
      created = order["Created"] != null ? ((DateTime)order["Created"]).ToString() : errString;
      // rest of the fields
}

Step 4:

HttpContext.Current.Response.Write(bld.ToString());

Now, when we have run step 2, what happens? Is the DataTable fully loaded?

I could see a problem if we poll the database in each and every row in step 3, as that could slow our process. But if it loads it all at once, I can’t see how this could be a problem.

So basically: does the DataTable load everything at once, or does it use lazy loading?

Answer 1

when we have run step 2, what happens? Is the datatable fully loaded?

All the values returned by the query will be transferred to the application and loaded into the DataTable.

does it use lazy loading?

No, it doesn’t.

---

What you can do:

• Consider a bounded query – only return the top 100 rows (or whatever is applicable). If there are thousands, it is unlikely to be of use.
• With the above, consider using paging to only fetch and display a bounded result set.
• Optimize the query. That it runs fast in testing, doesn’t mean the query is optimal. You may want to test this on a similar amount of data the your client has.