Question

When exporting a lot of data to a string (csv format), I get a OutOfMemoryException. What’s the best way to tackle this? The string is returned to a Flex Application.

What I’d do is export the csv to the server disk and give back an url to Flex. Like this, I can flush the stream writing to the disk.

Update:

String is build with a StringBuilder:

StringBuilder stringbuilder = new StringBuilder();
string delimiter = ";";
bool showUserData = true;

// Get the data from the sessionwarehouse
List<DwhSessionDto> collection 
     =_dwhSessionRepository.GetByTreeStructureId(treeStructureId);

// ADD THE HEADERS
stringbuilder.Append("UserId" + delimiter);
if (showUserData)
{
    stringbuilder.Append("FirstName" + delimiter);
    stringbuilder.Append("LastName" + delimiter);
}
stringbuilder.Append("SessionId" + delimiter);
stringbuilder.Append("TreeStructureId" + delimiter);
stringbuilder.Append("Name" + delimiter);
stringbuilder.Append("Score" + delimiter);
stringbuilder.Append("MaximumScore" + delimiter);
stringbuilder.Append("MinimumScore" + delimiter);
stringbuilder.Append("ReducedScore" + delimiter);
stringbuilder.Append("ReducedMaximumScore" + delimiter);
stringbuilder.Append("Duration" + delimiter);
stringbuilder.AppendLine("Category" + delimiter);

foreach (var dwhSessionDto in collection)
{
    stringbuilder.Append(
        getPackageItemsInCsvFromDwhSessionDto(
            dwhSessionDto, delimiter, showUserData));
}

return stringbuilder.ToString();

The string is sent back to Flex like this:

var contentType = "text/csv";
string result = exportSessionService.ExportPackage(treeStructureId);
// Write the export to the response
_context.Response.ContentType = contentType;
_context.Response.AddHeader("content-disposition", 
    String.Format("attachment; filename={0}", treeStructureId + ".csv"));

// do not Omit the Vary star so the caching at the client side will be disabled
_context.Response.Cache.SetOmitVaryStar(false);
_context.Response.Cache.SetCacheability(HttpCacheability.NoCache);

if (!String.IsNullOrEmpty(result))
{
    _context.Response.Output.Write(result);
    _context.Response.Output.Close();
}
else
{
    _context.Response.Output.Write("No logs");
}

Answer 1

String is created with a StringBuilder and communicated back to Flex Application with HttpContext.Reponse.Output.Write(result); – Lieven Cardoen

So what you’re basically doing is this:

StringBuilder str = new StringBuilder();
while(whatever)
{
    str.Append(thisNextLineIGuess);
}
Response.Output.Write(str.ToString());

Correct? That may not be your exact code, but it sounds like a good approximation.

Then the solution is simple:

while(whatever)
{
    Response.Output.Write(thisNextLineIGuess);
}

Streaming is usually better than buffering the whole thing and sending it out as a lump.