Question

I have suddenly been getting the memory exception errors for two programs running on different machines and even though it seems there is enough memory, it still shows up. I am creating multiple threads in the program so not sure if this appropriate for this forum but could it be something else related to visual studio or is it definitely memory issue.The one program runs on my desktop using visual studio 2008 with 2 gb ram. The other is running on a windows 2003 server with 4 GB ram using visual basic 2008 express.
Now the module takes a large xml file that is read into a string and then split and stored in a string array. Now the number of chunks can be upto 10000. Now I know this is big, but I have been running this for over a month now and never had the issue. The only other possible related issue I noticed was that I was running out of space on my harddrive but that was quickly solved with a cleanup. Oh yes the processor for my machine is a duo core set at 2.13 GHZ.
It is a console program that makes multiple webrequests but the memory problem arises in one specific module only as I explained above.

Public Shared Function textLoad(ByVal _html As String) As Boolean
    Try 
            //_html is the filestream that was read in

        Dim defaultHeading = "xmlns:gnip=""http://www.gnip.com/schemas/2010"" xmlns=""http://www.w3.org/2005/Atom"""
        Dim header_of_xml As String = "<?xml version=""1.0"" encoding=""utf-8""?>" & vbNewLine & "<entry " & defaultHeading & ">"
        Dim footer_of_xml As String = "</entry>"
        Dim entry As String = String.Empty
        Dim xrs As XmlReaderSettings = New XmlReaderSettings()
        Dim dupeArray As New ArrayList
        Dim stringSplitter() As String = {"</entry>"}

        //split the file content based on the closing entry tag
        sampleResults = Nothing
        sampleResults = _html.Split(stringSplitter, StringSplitOptions.RemoveEmptyEntries)
        entryResults.Clear()

        If getEntryId(sampleResults) Then
           // the following loops seem clumsy but I wanted to dedupe the lists to //ensure that I am not adding duplicate entries and I do this by going to the getEntryID //method and extracting the ids and then comparing them below 

            For i As Integer = 0 To sampleResults.Count - 1
                For j As Integer = 0 To idList.Count - 1
                    If sampleResults(i).Contains(idList.Item(j)) AndAlso Not dupeArray.Contains(idList.Item(j)) Then
                        dupeArray.Add(idList.Item(j))
                        entry = sampleResults(i)

I did look at taskmanager for identifying resources used by this program and this is what is going on:

Parser.exe CPU = 34
MEM usage = 349,500 K

nothing else intensive is running

Edit _–

Figured out exactly where the problem originates:

**sampleResults = _html.Split(stringSplitter, StringSplitOptions.RemoveEmptyEntries)**

Can anyone notice anything wrong with this??

Answer 1

The Split method allocates memory for a string object for each element in the returned array. If you’re going to run into a memory issue, it’s going to be on that line. Since you’re allocating memory for potentially 10,000 large strings at once, on a potentially fragmented heap, it’s not surprising that you might come to a point where it just can’t find enough contiguous space to allocate for the next string (which would result in the exception you are getting). Do you really need all of those strings at once? Or can you do something like this loop:

1. Read one line at a time (or into a buffer) to build a string.
2. Stop when you’ve found the first “chunk”.
3. Get the info you need from the chunk. (I’m assuming you don’t need the whole thing, you’re just interested in getting something out of it?).
4. Continue reading the html (Restart at #1), reading it into the same string variable so that the first chunk can get garbage collected.

If you can implement your solution that way, you’re issue should disappear.