I am designing a web API which requires fast read only access to a large dataset which will be hopefully be constantly stored and ready for access. Access will be from a static class which will just do some super fast lookups on the data.
So, I want to pre-cache a Dictionary<string,Dictionary<string,Dictionary<string,myclass>>>, with the total number of elements at the third level dictionary being around 1 Million, which will increase eventually, but lets say not more than 2 million ever. ‘myclass’ is a small class with a (small) list of strings, an int, an enum and a couple of bools, so nothing major. It should be a bit over 100mb in memory.
From what I can tell, the way to do this is simply call my StaticClass.Load() method to read all this data in from a file with the Application_Start event in Global.asax.
I am wondering what the things I need to consider/worry about with this. I am guessing it is not just as simple as calling Load() and then assuming everything will be OK for future access. Will the GC know to leave the data there even if the API is not hit for a couple of hours?
To complicate things, I want to reload this data every day as well. I think I’ll just be able to throw out the old dataset and load the new one in from another file, but I’ll get to that later.
Cheers
Please see my similar question IIS6 ASP.NET 2.0 Application Cache – data storage options and performance for large amounts of data BUT in particular the answer from Marc and his last paragraph about options for large cache’s which I think would apply to your case.