Question

I’m constantly (every 30-60 min) getting a System.OutOfMemoryException in my Windows Service. The service’s job is to loop though 6 directories which contains data files which the service datawashes to a common XML data format.

These 6 folders contains 5-10.000 files each, so the total number of files is about 45.000 and new files is added duing the day. There is added about 1-2000 new files a day. The files is between 4KB and 500KB.

Each data file is washed to the common XML data format through the XElement object.

I have used RedGates ANTS Memory Profiler on the service and the objects which are using the most memory is string (about 90.000.000 bytes) and XElement (about 51.000.000 bytes).

In the Memory Profiler, when i trace, what is using the string object, i can see that it’s mostly (93%) the XElement object which is using the string object.

The server have 6 cpu’s and 6GB of RAM, so i can’t see why i’m getting the OutOfMemoryException. If i look at the Windows Service in the Processes it’s MAX use of RAM have been 1.2GB.

I have read that .NET garbage collector doesn’t clear the string object because the string object is stored in a intern table. Could this be the error, if so what can i do about it?

The code below shows how i’m looping through the files. As you can see i have also tried to take 20 files at a time. This just pushes the OutOfMemoryException a few hours, so the service will run for 4-5 hours instead of 30-60 min.

Why do i can the OutOfMemoryException?

private static void CheckExistingImportFiles(object sender, System.Timers.ElapsedEventArgs e)
    {
        CheckTimer.Stop();
        var dir = Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories);

        List<ManualResetEvent> doneEvents = new List<ManualResetEvent>();
        int i = 0;
        //int doNumberOfFiles = 20;

        foreach (string existingFile in Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories))
        {
            if (existingFile.EndsWith("ignored") || existingFile.EndsWith("error") || existingFile.EndsWith("importing"))
            {
                //if (DateTime.UtcNow.Subtract(File.GetCreationTimeUtc(existingFile)).TotalDays > 5)
                //  File.Delete(existingFile);
                //continue;
            }

            StringBuilder fullFileName = new StringBuilder().Append(existingFile);

            if (!fullFileName.ToString().ToLower().EndsWith("error") && !fullFileName.ToString().ToLower().EndsWith("ignored") && !fullFileName.ToString().ToLower().EndsWith("importing"))
            {
                File.Move(fullFileName.ToString(), fullFileName + ".importing");
                fullFileName = fullFileName.Append(".importing");

                ImportFileJob newJob = new ImportFileJob(fullFileName.ToString());

                doneEvents.Add(new ManualResetEvent(false));

                ThreadPool.QueueUserWorkItem(newJob.Run, doneEvents.ElementAt(i));
                i++;
            }

            //if (i > doNumberOfFiles)
            //{
            //    i = 0;
            //    doNumberOfFiles = 20;
            //    break;
            //}
        }
        i = 0;
        WaitHandle.WaitAll(doneEvents.ToArray());

        CheckTimer.Start();
    }

Answer 1

As Avner Shahar-Kashtan already stated, I also think that the problem is in ImportJob(you haven’t shown us its code).

Even so, you can still make some optimizations.

You don’t have to load all file names at once. It can be done dir by dir as below

IEnumerable<string> GetAllFiles(string dirName)
{
    var dirs = Directory.GetDirectories(dirName);

    foreach (var file in Directory.GetFiles(dirName))
        yield return file;

    foreach (var dir in dirs) //recurse
        foreach (var file in GetAllFiles(dir)) 
            yield return file;
}

And by using TPL, you can reduce the number of ManualResetEvents created (and their forgotten Dispose()s)

Parallel.ForEach(GetAllFiles(RawDataDirectory.FullName) , file =>
{
    //ImportFileJob newJob = new ImportFileJob(file);
    //newJob.Run
    Console.WriteLine(file);
}); 

BTW, you should also see CountdownEvent