Question

I am experiencing problem in my application with OutOfMemoryException. My application can search for words within texts. When I start a long running process search to search about 2000 different texts for about 2175 different words the application will terminate at about 50 % through with a OutOfMemoryException (after about 6 hours of processing)

I have been trying to find the memory leak. I have an object graph like: (–> are references)

a static global application object (controller) –> an algorithm starter object –> text mining starter object –> text mining algorithm object (this object performs the searching).

The text mining starter object will start the text mining algorithm object’s run()-method in a separate thread.

To try to fix the issue I have edited the code so that the text mining starter object will split the texts to search into several groups and initialize one text mining algorithm object for each group of texts sequentially (so when one text mining algorithm object is finished a new will be created to search the next group of texts). Here I set the previous text mining algorithm object to null. But this does not solve the issue.

When I create a new text mining algorithm object I have to give it some parameters. These are taken from properties of the previous text mining algorithm object before I set that object to null. Will this prevent garbage collection of the text mining algorithm object?

Here is the code for the creation of new text mining algorithm objects by the text mining algorithm starter:

    private void RunSeveralAlgorithmObjects()
    {

        IEnumerable<ILexiconEntry> currentEntries = allLexiconEntries.GetGroup(intCurrentAlgorithmObject, intNumberOfAlgorithmObjectsToUse);

        algorithm.LexiconEntries = currentEntries;
        algorithm.Run();

        intCurrentAlgorithmObject++;

        for (int i = 0; i < intNumberOfAlgorithmObjectsToUse - 1; i++)
        {
            algorithm = CreateNewAlgorithmObject();
            AddAlgorithmListeners();
            algorithm.Run();
            intCurrentAlgorithmObject++;
        }

    }

    private TextMiningAlgorithm CreateNewAlgorithmObject()
    {
        TextMiningAlgorithm newAlg = new TextMiningAlgorithm();

        newAlg.SortedTermStruct = algorithm.SortedTermStruct;
        newAlg.PreprocessedSynonyms = algorithm.PreprocessedSynonyms;
        newAlg.DistanceMeasure = algorithm.DistanceMeasure;
        newAlg.HitComparerMethod = algorithm.HitComparerMethod;
        newAlg.LexiconEntries = allLexiconEntries.GetGroup(intCurrentAlgorithmObject, intNumberOfAlgorithmObjectsToUse);
        newAlg.MaxTermPercentageDeviation = algorithm.MaxTermPercentageDeviation;
        newAlg.MaxWordPercentageDeviation = algorithm.MaxWordPercentageDeviation;
        newAlg.MinWordsPercentageHit = algorithm.MinWordsPercentageHit;
        newAlg.NumberOfThreads = algorithm.NumberOfThreads;
        newAlg.PermutationType = algorithm.PermutationType;
        newAlg.RemoveStopWords = algorithm.RemoveStopWords;
        newAlg.RestrictPartialTextMatches = algorithm.RestrictPartialTextMatches;
        newAlg.Soundex = algorithm.Soundex;
        newAlg.Stemming = algorithm.Stemming;
        newAlg.StopWords = algorithm.StopWords;
        newAlg.Synonyms = algorithm.Synonyms;
        newAlg.Terms = algorithm.Terms;
        newAlg.UseSynonyms = algorithm.UseSynonyms;

        algorithm = null;

        return newAlg;
    }

Here is the start of the thread that is running the whole search process:

            // Run the algorithm in it's own thread
            Thread algorithmThread = new Thread(new ThreadStart
                (RunSeveralAlgorithmObjects));
            algorithmThread.Start();

Can something here prevent the previous text mining algorithm object from being garbage collected?

Answer 1

I recommend first identifying what exactly is leaking. Then postulate a cause (such as references in event handlers).

To identify what is leaking:

1. Enable native debugging for the project. Properties -> Debug -> check Enable unmanaged code debugging.

2. Run the program. Since the memory leak is probably gradual, you probably don’t need to let it run the whole 6 hours; just let it run for a while and then Debug -> Break All.

3. Bring up the Immediate window. Debug -> Windows -> Immediate

4. Type one of the following into the immediate window, depending on whether you’re running 32 or 64 bit, .NET 2.0/3.0/3.5 or .NET 4.0:

   .load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll for 32-bit .NET 2.0-3.5

   .load C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\sos.dll for 32-bit .NET 4.0

   .load C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\sos.dll for 64-bit .NET 2.0-3.5

   .load C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\sos.dll for 64-bit .NET 4.0

5. You can now run SoS commands in the Immediate window. I recommend checking the output of !dumpheap -stat, and if that doesn’t pinpoint the problem, check !finalizequeue.

Notes:

• Running the program the first time after enabling native debugging may take a long time (minutes) if you have VS set up to load symbols.
• The debugger commands that I recommended both start with ! (exclamation point).

These instructions are courtesy of the incredible Tess from Microsoft, and Mario Hewardt, author of Advanced .NET Debugging.

Once you’ve identified the leak in terms of which object is leaking, then postulate a cause and implement a fix. Then you can do these steps again to determine for sure whether or not the fix worked.