Question

I have a large DataTable cached in my web app that is the result of a complex query that returns a large data set. Whilst this data table is cached the query that runs to “refresh” this cache still takes a long time, largely due to the sheer amount of data being returned.

In order to speed this up I am considering implementing a timestamp type approach to my tables in order to limit my query to only return rows which have changed.

I then intend to merge this smaller dataset with my cached datatable.

Has anyone done anything similar to this, or is there anything out there that handles this already?

I feel this could be a re-inventing the wheel situation if I dive straight in.

Answer 1

Personally, I’ve used the timestamp approach before and that does work well – it does make the caching more efficient by only retrieving the data that has changed since the last read.

Alternatively, I’d suggest the SqlCacheDependency class which takes care of keeping the cache up to date for you. I can’t comment on any real-world pros + cons of this, or of performance comparison vs. timestamp approach as I haven’t used it myself.

There’s another useful article on SqlCacheDependency here

Update:
Yes, I don’t think it will actually refresh the data. It sounds like you’d have to do that yourself.
From the 2nd link:

When the data changes—and only
then—the cache items based on that
data are invalidated and removed from
the cache. The next time you request
that item from the cache, if it is not
in the cache, you can re-add the
updated version to the cache and be
assured that you have the latest data

There’s also SQL 2005 specific implementation notes in the 2nd link:

SQL Server 2005 monitors changes to
the result set of a particular SQL
command. If a change occurs in the
database that would modify the results
set of that command, the dependency
causes the cached item to be
invalidated. This allows SQL Server
2005 to provide row-level
notification.

I personally think I’d go for the timestamp approach (that’s what I’ve done before) as I can’t see on the face of it that SqlCacheDependency would give any performance benefits – I think it would be less performant (just easier to implement). One day, I’ll get round to actually trying out SqlCacheDependency to do a proper performance analysis 🙂

Update 2:
Regarding the merging of new data into the existing datatable, I think the Merge method of the datatable is what you want.

The Merge method is used to merge two DataTable objects that have largely similar schemas. A merge is typically used on a client application to incorporate the latest changes from a data source into an existing DataTable.
…
…
When merging a new source DataTable into the target, any source rows with a DataRowState value of Unchanged, Modified, or Deleted, is matched to target rows with the same primary key values. Source rows with a DataRowState value of Added are matched to new target rows with the same primary key values as the new source rows.

You just need to ensure you define the column(s) on the datatable that are the primary key.