Question

I have a console application which runs hundred times a day and reads same data from large files (say 50 files with total size of 3-4 Gb).

I am thinking of making a Windows service which will cache the data in memory to speed the access and also control life time of the data (as set of those 50 files may vary from day to day).

I am going to implement shared memory mapped files so the console application will read files from memory written by the service…

However there is another consideration. The data read is converted to .NET objects every time.

So my question – is there a way to share not files but .NET objects (List) in memory?

P.S. the data is series of bytes serialised manually

Answer 1

I would just implement this as a service that provides a socket-server, and have the console exe just connect to the service and make a request over TCP/IP, getting the response back. Pretty easy to setup, since you already (per the question) have serialization sorted, and very scalable. The service can then keep everything running happily in memory. You can even have the client and server be the same exe – just check Environment.UserInteractive when it starts to decide whether to be a client vs a server.

One note: keep the bandwidth low, and avoid a chatty API; meaning: don’t have the client do lots of thinking then make 200 requests; just have it package up the entire request and ship it to the server. Let the server worry about that all locally, avoiding lots of network traffic.