Question

I’m working with an architecture that requires mixing large (static) tables joined with very frequently changing data.

To emphasize the point, imagine SO website user access inner joined to their user database).

SELECT * FROM UserProfile INNER JOIN OnlineUser (on UserProfiles.id = OnlineUser.id)

where UserProfile reside in a large SQL table and OnlineUser is dynamic data on the Webserver.

Putting everything memory takes up a lot of room and putting everything in the database would really tax the server (shudder to think of it). Is there a better way of doing this?

God Jon Skeet says LINQ can’t cope with doing a join between an in-memory collection and a database table. He suggests a contains clause or list, both of which wouldn’t be appropriate in this case.

Edit:

An in-memory table (PINTABLE) in SQL Server could do this. Since that feature has been deprecated, is it safe to assume SQL server 2008 will figure out to keep it in memory to reduce IO?

Answer 1

You have to do the actual join operation somewhere. Either you bring the OnlineUser data to the SQL Server and perform the join there, or you bring the UserProfile data into memory and perform the join there.

Usually it’s better to let the database server sort out the data needed and only read that into memory. One way to solve the problem is to create a temp table on the SQL Server, where you put the data fields required for the join operation (in your example OnlineUser.id).

Then execute a SQL query to get the data required into memory. Make sure that you have indexes that speeds up the filtering. Once retrieved to memory, match the two collections (e.g. by having them sorted on the same key and then use the linq Zip operator)