Question

I have some large csv files (1.5gb each) where I need to replace specific values. The method I’m currently using is terribly slow and I’m fairly certain that there should be a way to speed this up but I’m just not experienced enough to know what I should be doing. This is my first post and I tried searching through to find something relevant but didn’t come across anything. Any help would be appreciated.

My other thought would be to break the file into chunks so that I can read the entire thing into memory, do all of the replacements there and then output to a consolidated file. I tried this but the way I did it actually ended up seeming slower than my current method.

Thanks!

    Sub Main()
    Dim fName As String = "2009.csv"
    Dim wrtFile As String = "2009.1.csv"
    Dim lRead
    Dim lwrite As String
    Dim strRead As New System.IO.StreamReader(fName)
    Dim strWrite As New System.IO.StreamWriter(wrtFile)
    Dim bulkWrite As String

    bulkWrite = ""
    Do While strRead.Peek <> -1
        lRead = Split(strRead.ReadLine(), ",")
        If lRead(9) = "5MM+" Then lRead(9) = "5000000"
        If lRead(9) = "1MM+" Then lRead(9) = "1000000"

        lwrite = ""
        For i = LBound(lRead) To UBound(lRead)
            lwrite = lwrite & lRead(i) & ","
        Next
        strWrite.WriteLine(lwrite)
     Loop

    strRead.Close()
    strWrite.Close()
End Sub

Answer 1

You are splitting and the combining, which can take some time.

Why not just read the line of text. Then replace any occurance of “5MM+” and “1MM+” with the approiate value and then write the line.

 Do While ...
    s = strRead.ReadLine();
    s = s.Replace("5MM+", "5000000")
    s = s.Replace("1MM+", "1000000")
    strWrite(s);
 Loop