Question

Here’s the background, I have a webservice from a vendor/partner that returns a large XML document that I use to display data. Simple enough. The problem is that the service takes several minutes to execute. The vendor is returning an entire normalized database as an XML document in one method call and it’s slow. Stupid vendor but I can’t control that.

I have two questions:

1. Where do I cache it? Disk, database, memory (I’m leaning to putting it on disk. It’s too big and not accessed enough to store in memory. Putting it in a DB is probably right but I don’t want to write ETL and have a job running somewhere.)

2. How do I cache it? I can’t just let it expire and have the next request refresh the cache because it takes so long. I need to serve stale data while the cache is rebuilding. I can think of a couple of ways of doing it but nothing that seems simple or very elegant. I’m looking to do something simple; I’m not looking for the best engineered solution ever.

Here’s my plan…please tell me that I’m an idiot and suggest something easier that I’m overlooking.

Write the XML to disk
At the same time write the filename and some type of expiry bomb into cache (obviously the filename would not expire for a very long time)
In my worker thread, try to read my bomb from cache
If the cache is expired (my bomb went off) then spawn a thread to get and save the XML
meanwhile, take the filename from cache and read in the old XML while the new one builds
when the new one is finished, expire the old filename from cache and write the filename to cache

This sounds pretty much ridiculous to me and there has to be a better way.

I’m using .Net and IIS6 and I can stick the XML in SQL Server if needed.

Answer 1

You could create a new database and a scheduled job to periodically fetch those data and store them into the database. Then you can serve the data that are actually needed directly from your database instead of web service.