Question

So, I have this code that grabs a bunch of data from the database, and performs a bunch of calculations on that data. However, this is not what is causing the halt. The halt is coming in when I take all that “final” data that has been prepared, and go to write that into a text file.

Each line in the text file is being created from data that has had calculations performed on it. Each line “corresponds” to a record from the database. A user will have to perform this file export usually on about 30-40 thousand records at a time. Now, even though I am using a StringBuilder, this still halts because of the pure mass amount of records.

What I do currently in the code is create one StringBuilder, append all the data (after calcs) to it, convert it to a String, add it to a List(Of String) to be returned from the Function, and then clear the StringBuilder, and do the same for the next row, and I do this until each row has been created and appended, converted, and then inserted into the List.

Then, another class receives this List(Of String) and takes each String and eventually appends it to a file.

Does anyone have an idea of how to improve this code and make it so it doesn’t halt the server every time? The reason it wasn’t found out until now is because when I tested it locally I was testing it on only 1-50 records at a time.

Thanks ahead of time!

Edit:

r = Class.ExportFile(ID)
data = Encoding.ASCII.GetBytes(r.ResponseString)

Return File(data, "text/plain", r.DefaultFileName)

-Scott

Answer 1

Return an IEnumerable instead of a big string or a list of strings and write a line at a time to the file. That way you can evaluate the data lazily, so you don’t need to keep huge strings around. e.g.

IEnumerable<string> Process() {
   var rows = QueryTheDatabase();
   foreach(var row in rows) {
      yield return ProcessARecord(row);//process and build a string of
                                         //one row
    }
 }

The caller just does a foreach over the returned IEnumerable and writes each line to a file.