I am moving a complex process out of SQL to a .NET application. I’m kind of taking a brute force approach by pulling down only the data that is needed from SQL, then storing in datatables. Using a pipeline pattern with stepping, I broke out the processes that can be done in parallel (not dependent upon the other processes, nor working on the same data bits).
Everything is going fine, but I want to know if there is an in-memory sql solution that would perform better than the DataSet/DataTable structures. We’re talking about 50k rows at a time with up to 1m supporting data rows (read 5b rows). Row size for 1 row (with all supporting data rows) is probably around 1K avg (due to large strings).
My question is specifically on the performance of DataSets, memory overhead and persistence. I will need to serialize the data at each stage to disk for recovery purposes.
Would it be better to just map the rows to a strongly typed model instead? I don’t need any relationships or other benefits of datasets; I replaced most of the search functionality with my own using parallel processing.
Data only uses primitive types, no blobs, streams, geography etc.
For local SQL operations, take a look at SQLite. I don’t recall offhand if it can run exclusively in memory, but with disk caching and transaction journalling turned off it would probably be pretty close.
Steve Shaunessey developed a fast in-memory SQL engine at Borland many years ago. I don’t know if it was ever productized. Take a look around the Embarcadero.com web site to see if any remnants of his work have survived.
I noticed aidaim.com advertises an in-memory SQL engine. No experience with it, just FYI.
Also consider LINQ for in-memory query operations. If you pay attention to what you’re doing, LINQ’s query composition and deferred execution work well with large data, IMO. And, no SQL parser required.