Question

I have a .net winform c# program.
I collect traces on my program, with my own trace system. The trace show information about what the user did and other usefull information.
I store each trace in a dataset. An example, I add a row in the dataset every second.
The programe dispose the traces only when the program encounter a undhandler exception or when the program close.

The problem is : The dataset begin to be very large in the memory, and if the program is running enough time, a out of memory exception occur.

I search the web and i found some ideas.
– put the dataset in a file on hard drive ( in this case I must open a big file each time i want to add a trace)
– paging the dataset

I must find a way to free the memory use by this dataset by keeping all the information it containts and also keep a way to add additionnal rows to this dataset.

Someone have ideas or guidance ? Thank you

Answer 1

You could save the content of the dataset to disk (every hour in its own file), clear the in-memory data, run for an hour in-memory, save the data into a separate file and so on.

This way your memory consumption should be ok and your performance gets only hit once an hour.

Since the files on disk all have the same structure they can be merged whenever you need the “complete data”.

Although I would recommend using some existing logging library – there are lots of well-tested one out there (free and commercial).