Question

Possible Duplicate:
How to efficiently write to file from SQL datareader in c#?

I am currently trying to create a web application that uses read-only access to allow users to download large files from our database. The table in question has 400,000 records in it and generates a 50 MB .csv file when exported.

It takes about 7s to run the statement “SELECT * FROM [table]” on SQL server, and about 33s to do so from my web application (hosted on a different server). This is reading all the data into a System.Data.SqlClient.SqlDataReader object.

My problem is that I am at a loss for converting my SqlDataReader to a .csv file. Converting each row of the SqlDataReader to a string and outputting that string to a file line by line takes almost 2 hours, which is unacceptable. Below is the code I’m using to create a file on the web application’s server:

    while (rdr.Read())
    {
        string lineout = "";
        for (int index = 0; index < rdr.FieldCount; index++)
            lineout += rdr[index].ToString().Replace(',', ' ') + ',';
        write(lineout, filename); //uses StreamWriter.WriteLine()
    }

There has to be a better way. I’ve looked around and saw a lot of suggestions that essentially recommend doing the above to create a file. This works great with smaller tables, but not the two really large ones we use every day. Can anyone give me a push in the right direction?

Answer 1

You could try building your lineout with a StringBuilder rather than manually concatenating strings:

//you can test whether it makes any difference in performance declaring a single
//StringBuilder and clearing, or creating a new one per loop
var sb = new StringBuilder();

while (rdr.Read())
{
    for (int index = 0; index < rdr.FieldCount; index++)
        sb.Append(rdr[index].ToString().Replace(',', ' ').Append(',');

    write(sb.ToString(), filename); //uses StreamWriter.WriteLine()
    sb.Clear();
}

Alternatively try to just write to the file directly and avoid generating each line in memory first:

//assume a StreamWriter instance has been created called sw...
while (rdr.Read())
{
    for (int index = 0; index < rdr.FieldCount; index++)
    {
        sw.Write(rdr[index].ToString().Replace(',', ' ');
        sw.WriteLine(",");
    }
}

//flush and close stream