Question

I have an application that generates 50,000 records containing certain information. Once generated and saved to the database, the user can download the records as csv any time they like from a control panel.

The problem I am getting is that it can take anything up to 40+ seconds to generate the CSV. But by the time its finished processing, the browser no longer tries to download it. Instead it just hangs.

I have currently set a high timeout value so the processing won’t timeout. Also I need to it be generated on the fly as it needs to contain related live data in with it.

Does anyone know why it doesn’t download after waiting for the file to generate?

Heres the headers I respond with once generated:

//**generate the csv as string here, and then...**
Response.Clear();
Response.AddHeader("content-disposition", string.Format("attachment;filename=file_{0}.csv", batchId));
Response.ContentType = "text/csv";
Response.Write(csv);
Response.End();

Answer 1

Don’t generate the entire csv file up front. Write records to the stream as you retrieve them from the database. You want code that looks something like this:

Response.Clear();
Response.AddHeader("content-disposition", string.Format("attachment;filename=file_{0}.csv", batchId));
Response.ContentType = "text/csv";

using (SqlDataReader rdr= GetDataForCSV())
{
    int i = 0;
    while (rdr.Read())
    {
       Response.Write(CSVFromIDataRecord(rdr));
       Response.Write("\n");
       if (i % 50 == 0)
       {
           //flush the output stream every now and then
           Response.Flush();
       }
    }
    rdr.Close();
}
Response.End();

Note that I don’t advocate this exact code. For example, there are some issues using an SqlDataReader in this way that fall outside the scope of the question. You’ll also want to set up the number of records between each flush call such that you take good advantage of your IIS server’s response buffer, depending on your specific average record size. I would make some other changes in production as well, but this should be enough to point you in the right direction.