Question

We have a website which is using linq to entities, we found that it’s very slow recently, after troubleshooting, I found whenever we use linq to entities to search data from database, it will consume very much CPU time, like toList() function. I know it might because we have lots of data in database which caused to slow response, but I just wonder if there are any other reasons which might cause this problem?

How should I do to optimize these kinds of problem? following is the possible reasons:

1. ToList() might load all object’s foreign object(foreign key), how can I force it only load the object?

2. Is my connection pool too small?

Please let me know if there are any other possible reason, and point me the right direction to solve this issue.

Answer 1

In Linq – a query returns the results from a sequence of manipulations to sources when the query is enumerated.

IQueryable<Customer> myQuery = ...

foreach(Customer c in myQuery)  //enumerating the query causes it to be executed
{

}

List<Customer> customers = myQuery.ToList();
  // ToList will enumerate the query, and put the results in a list.
  // enumerating the query causes it to be executed.

An executing query requires a few things (in no particular order)

• A database connection is drawn from the pool.
• The query is interpreted by the query provider (in this case, the provider is linq to entities and the interpretation is some form of sql)
• The interpretted form is transmitted to the database, where it does what it does and returns data objects.
• Some method must be generated to translate the incoming data objects into the desired query output.
• The database connection is returned to the pool.
• The desired query output may have state tracking done to it before it is returned to your code.

Additionally, the database has a few steps, here listed from the point of view of querying a sql server:

• The query text is recieved and checked against the query plan cache for an existing plan.
• If no plan exists, a new one is created and stuck into the plan cache by the query optimizer.
• The query plan is executed – IO/locks/CPU/Memory – any of these may be bottlenecks
• Query results are returned – network may be a bottleneck, particularly if the resultset is large.

So – to find out where the problem with your query is, you need to start measuring. I’ll order these targets in the order I’d check them. This is not a complete list.

1. Get the translated sql text of the query. You can use sql server profiler for this. You can use the debugger. There are many ways to go about it. Make sure the query text returns what you require for your objects, no more no less. Make sure the tables queried match your expectations. Run the query a couple times.

2. Look at the result set. Is it reasonable or are we looking at 500 Gigs of results? Was a whole table queried, when the whole thing wasn’t needed? Was a cartesian result generated unexpectedly?

3. Get the execution plan of the query (in sql studio, click the show estimated execution plan button). Does the query use the indexes you expect it to? Does the plan look wierd (possibly a bad plan came from the cache)? Does the query work on tables in the order you expect it to, and perform nested/merge/hash joins in the way you expect? Is there parallellization kicking in, when the query doesn’t deserve it (this is a sign of bad indexes/TONS of IO)?

4. Measure the IO of the query. (in sql server, issue SET STATISTICS IO ON). Examine the logical IO per table. Which table stands out? Again, look for a wrong order of table access or an index that can support the query.

   If you’ve made it this far, you’ve likely found and fixed the problem. I’ll keep going though, in case you haven’t.

5. Compare the execution time of the query to the execution time of the enumeration. If there’s a large difference, it may be that the code which interprets the data objects is slow or that it generated slow. It could also be that the translation of the query took a while. These are tricky problems to solve (in LinqToSql we use compiled queries to sort them out).

6. Measure Memory and CPU for the machine the code is running on. If you are capped there, use a code profiler or memory profiler to identify and resolve the issue.

7. Look at the network stats on the machine, in particular you may want to use TCPView to see the TCP socket connections on the machine. Socket resources may be mis-used (such as opening and closing thousands in a minute).

8. Examine the database for locks held by other connections.

I guess that’s enough. Hope I didn’t forget any obvious things to check.