Question

I want to know something, I have this loop that runs for all days (7 times) and then another loop inside it, that runs for all the records in the file. (about 100000), so all in all its about a 700000 times, now I want to log each processing in each loop, and log that to a file, say we are inside 1st loop for first time, and 2nd loop for first time, we log each time, what is done in a file. But the problem is that if I were to log each time, it would terribly hurt the performance, because of so many IO operations, what I was thinking is that is there any way, I could log each and every step to memory (memory stream or anything) and then at the end of outer loop, log all that memory stream data to file?

say if I have

for (int i=0; i<7; i++)
{
  for (int j=0; j<RecordsCount; j++)
  {
    SomeOperation();
    // I am logging to a file here right now
  }
}

Its hurting the performance terribly, if I remove the logging file, I could finish it a lot earlier. So I was thinking it would be better to log to memory first and write all that log from memory to file. So, is there any way to do this? If there are many, what’s the best?

ps: here is a logger I found http://www.codeproject.com/Articles/23424/TracerX-Logger-and-Viewer-for-NET, but I can use any other custom logger or anything else if required.

EDIT : If I use memory stream, and then write it all to file, would that give me better performance then using File.AppendAllLines as suggested in answer by Yorye and zmbq, also if would that give me any performance gain over what Jeremy commented?

Answer 1

Why not use a proper logging framework instead of writing your own?

NLog for instance has buffering built in and it is easy to configure: https://github.com/nlog/NLog/wiki/BufferingWrapper-target

I suggest you focus on writing code that gives value to your project while reusing existing solutions for all the other stuff. That will probably make you more efficient and give better results 🙂