Question

I’ve a console application where I need to read a few .csv files.
I’ve no problem to read the two first wich are quite ligth (about 10 000 records each).
But when it start reading the third one (way heavier, about 220 000 records) I keep having error “OutOfMemoryException was unhandled: An unhandled exception of type ‘System.OutOfMemoryException’ occurred in mscorlib.dll”

I don’t understand seen it s working fine for the two first csv…

Here’s my function reading the csv files, the error occurs on the split:

Function FileToString(filePath As String) As String()
    Dim myfile As New StreamReader(filePath, System.Text.Encoding.GetEncoding("iso-8859-1"))
    Dim allData As String = myfile.ReadToEnd()
    Dim rows As String() = allData.Split(vbCr.ToCharArray)
    Return rows
End Function

How could I prevent this? Is there a better way to read csv?
Thanx

Answer 1

There are 2 things to consider

1) Instead of String use StringBuilder as
Dim builder As New StringBuilder

2) For file reading use buffered approach instead of myfile.ReadToEnd()

Example

Function FileToString(filePath As String) As String 
    Dim f As System.IO.FileStream
    Dim mylength As Integer
    Dim i As Integer

    f = New System.IO.FileStreamfilePath, IO.FileMode.Open, IO.FileAccess.Read)

    Dim streamLength As Integer = Convert.ToInt32(f.Length)
    Dim fileData As Byte() = New Byte(streamLength) {}

    f.Read(fileData , 0, streamLength)
    f.Close()
    return fileData.ToString();
End Function 

And if you want line by line then use below code

Dim sr As StreamReader = New StreamReader("TestFile.txt")
Dim line As String
Do
    line = sr.ReadLine()
    Console.WriteLine(Line)
Loop Until line Is Nothing
sr.Close()