Question

I am working with twitter api data and after storing the stream results in text files, I input the data into a parser application. What I have planned for was large data files so I read the content in using delimiter]} to separate the individual posts to avoid the potential for errors? A backup function was to read the data using a buffer and then snip into individual posts.
But the problem is that in some cases for a single post, a memory exception will occur. Now when I look at the individual post it does not seem necessarily large but the text will contain foreign characters or some encoding I guess that causes the memory exception. I have not figured out if is exactly this yet but thought I would get some input or advice from here…

        myreader.TextFieldType = FileIO.FieldType.Delimited
        myreader.SetDelimiters("]}}")
        Dim currentRow As String()

        Try

            While Not myreader.EndOfData
                Try
                    currentRow = myreader.ReadFields()
                    Dim currentField As String

                    For Each currentField In currentRow
                        data = data + currentField
                        counter += 1
                        If counter = 1000 Then
                            Dim pt As New parsingUtilities
                            If Not data = "" Then
                                pt.getNodes(data)
                                counter = 0
                            End If
                        End If
                    Next
                Catch ex As Exception
                    If ex.Message.Contains("MemoryException") Then
                        fileBKup()
                    End If
                End Try

the other time when a memory exception occurs is then I try to split into different posts:

    Dim sampleResults() As String
    Dim stringSplitter() As String = {"}}"}

    ' split the file content based on the closing entry tag
    sampleResults = Nothing
    Try
        sampleResults = post.Split(stringSplitter, StringSplitOptions.RemoveEmptyEntries)

    Catch ex As Exception
        appLogs.constructLog(ex.Message.ToString, True, True)
        moveErrorFiles(form1.infile)
        Exit Sub
    End Try

Answer 1

I expect the problem is the strings.

Strings are immutable, meaning that every time you think you’re changing a string by doing this

data = data + currentField  

you’re actually creating another new string in memory. So if you do that thousands of times it can cause a problem because they mount up and you get an OutOfMemoryException.

If you’re building up strings you should use a StringBuilder instead.