Question

I have a situation where my application constructs a dynamic LINQ query using PredicateBuilder based on user-specified filter criteria (aside: check out this link for the best EF PredicateBuilder implementation). The problem is that this query usually takes a long time to run and I need the results of this query to perform other queries (i.e., joining the results with other tables). If I were writing T-SQL, I’d put the results of the first query into a temporary table or a table variable and then write my other queries around that. I thought of getting a list of IDs (e.g., List<Int32> query1IDs) from the first query and then doing something like this:

var query2 = DbContext.TableName.Where(x => query1IDs.Contains(x.ID))

This will work in theory; however, the number of IDs in query1IDs can be in the hundreds or thousands (and the LINQ expression x => query1IDs.Contains(x.ID) gets translated into a T-SQL “IN” statement, which is bad for obvious reasons) and the number of rows in TableName is in the millions. Does anyone have any suggestions as to the best way to deal with this kind of situation?

Edit 1: Additional clarification as to what I’m doing.

Okay, I’m constructing my first query (query1) which just contains the IDs that I’m interested in. Basically, I’m going to use query1 to “filter” other tables. Note: I am not using a ToList() at the end of the LINQ statement—the query is not executed at this time and no results are sent to the client:

var query1 = DbContext.TableName1.Where(ComplexFilterLogic).Select(x => x.ID)

Then I take query1 and use it to filter another table (TableName2). I now put ToList() at the end of this statement because I want to execute it and bring the results to the client:

var query2 = (from a in DbContext.TableName2 join b in query1 on a.ID equals b.ID select new { b.Column1, b.column2, b.column3,...,b.columnM }).ToList();

Then I take query1 and re-use it to filter yet another table (TableName3), execute it and bring the results to the client:

var query3 = (from a in DbContext.TableName3 join b in query1 on a.ID equals b.ID select new { b.Column1, b.column2, b.column3,...,b.columnM }).ToList();

I can keep doing this for as many queries as I like:

var queryN = (from a in DbContext.TableNameN join b in query1 on a.ID equals b.ID select new { b.Column1, b.column2, b.column3,...,b.columnM }).ToList();

The Problem: query1 is takes a long time to execute. When I execute query2, query3…queryN, query1 is being executed (N-1) times…this is not a very efficient way of doing things (especially since query1 isn’t changing). As I said before, if I were writing T-SQL, I would put the result of query1 into a temporary table and then use that table in the subsequent queries.

Edit 2:

I’m going to give the credit for answering this question to Albin Sunnanbo for his comment:

When I had similar problems with a heavy query that I wanted to reuse in several other queries I always went back to the solution of creating a join in each query and put more effort in optimizing the query execution (mostly by tweaking my indexes).

I think that’s really the best that one can do with Entity Framework. In the end, if the performance gets really bad, I’ll probably go with John Wooley’s suggestion:

This may be a situation where dropping to native ADO against a stored proc returning multiple results and using an internal temp table might be your best option for this operation. Use EF for the other 90% of your app.

Thanks to everyone who commented on this post…I appreciate everyone’s input!

Answer 1

If the size of TableName is not too big to load the whole table you use

var tableNameById = DbContext.TableName.ToDictionary(x => x.ID);

to fetch the whole table and automatically put it in a local Dictionary with ID as key.

Another way is to just “force” the LINQ evaluation with .ToList(), in the case fetch the whole table and do the Where part locally with Linq2Objects.

var query1Lookup = new Hashset<int>(query1IDs);
var query2 = DbContext.TableName.ToList().Where(x => query1IDs.Contains(x.ID));

Edit:
Storing a list of ID:s from one query in a list and use that list as filter in another query can usually be rewritten as a join.
When I had similar problems with a heavy query that I wanted to reuse in several other queries I always went back to the solution of creating a join in each query and put more effort in optimizing the query execution (mostly by tweaking my indexes).