Question

I have a task, which I know how to code (in C#),
but I know a simple implementation will not meet ALL my needs.
So, I am looking for tricks which might meet ALL my needs.

1. I am writing a simulation involving N number of entities interacting over time.

2. N will start at around 30 and move in to many thousands.

   a. The number of entities will change during the course of the simulation.
   

   b. I expect this will require each entity to have its own trace file.

3. Each Entity has a minimum of 20 parameters, up to millions; I want to track over time.

   a. This will most likely required that we can’t keep all values in memory at all times. Some subset should be fine.

   b. The number of parameters per entity will initially be fixed, but I can think of some test which would have the number of parameters slowing changing over time.

4. Simulation will last for millions of time steps and I need to keep every value for every parameter.

5. What I will be using these traces for:

   a. Plotting a subset (configurable) of the parameters for a fixed amount of time from the current time step to the past.

   i.  Normally on the order of 300 time steps.
   
   ii. These plots are in real time while the simulation is running.
   

   b. I will be using these traces to re-play the simulation, so I need to quickly access all the parameters at a give time step so I can quickly move to different times in the simulation.

   i.  This requires the values be stored in a file(s) which can be inspected/loaded after restarting the software.
   
   ii. Using a database is NOT an option.
   

   c. I will be using the parameters for follow up analysis which I can’t define up front so a more flexible system is desirable.

My initial thought:

1. One class per entity which holds all the parameters.

2. Backed by a memory mapped file.

3. Only a fixed, but moving, amount of the file is mapped to main memory

4. A second memory mapped file which holds time indexes to main file for quicker access during re-playing of simulation. This may be very important because each entity file will represent a different time slice of the full simulation.

Answer 1

KISS — just write a logfile for each entity and at each time slice write out every parameter in a specified order (so you don’t double the size of the logfile by adding parameter names). You can have a header in each logfile if you want to specify the parameter names of each column and the identify of the entity.

If there are many parameter values that will remain fixed or slowly changing during the course of the simulation, you can write these out to another file that encodes only changes to parameter values rather than every value at each time slice.

You should probably synchronize the logging so that each log entry is written out with the same time value. Rather than coordinate through a central file, just make the first value in each line of the file the time value.

Forget about database – too slow and too much overhead for simulation replay purposes. For replaying of a simulation, you will simply need sequential access to each time slice, which is most efficiently and fastest implemented by simply reading in the lines of the files one by one.

For the same reason – speed and space efficiency – forget XML.