Question

My application has to export the result of a stored procedure to .csv format. Basically, the client performs a query, and he can see the results on a paged grid, if it contains what he wants, then he clicks on a “Export to CSV” button and he downloads the whole thing.

The server will have to run a stored procedure that will return the full result without paging, create the file and return it to the user.

The result file could be very large, so I’m wondering what is the best way to create this file in the server on demand and serve it to the client without blow up the server memory or resources.

The easiest way: Call the stored procedure with LINQ, create a stream and iterate over the result collection and creating a line in the file per collection item.

1. Problem 1: Does the deferred execution applies as well to LINQ to stored procedures? (I mean, will .NET try to create a collection with all the items in the result set in memory? or will it give me the result item by item if I do an iteration instead of a .ToArray?)

2. Problem 2: Is that stream kept in RAM memory till I perform a .Dispose/.Close?

The not-so-easy way: Call the stored procedure with a IDataReader and per each line, write directly to the HTTP response stream. It looks like a good approach, as long as I read I write to the response, the memory is not blown up.

Is it really worth it?

I hope I have explained myself correctly.

Thanks in advance.

Answer 1

Writing to a stream is the way to go, as it will rougly consume not more than the current “record” and associated memory. That stream can be a FileStream (if you create a file) or the ASP.NET stream (if you write directly to the web), or any other useful stream.

The advantage of creating a file (using the FileStream) is to be able to cache the data to serve the same request over and over. Depending on your need, this can be a real benefit. You must come up with an intelligent algorithm to determine the file path & name from the input. This will be the cache key. Once you have a file, you can use the TransmitFile api which leverages Windows kernel cache and in general very efficient. You can also play with HTTP client caches (headers like last-modified-since, etc.), so next time the client request the same information, you may return a not modified (HTTP 304 status code) response. The disadvantages of using cache files is you will need to manage these files, disk space, expiration, etc.

Now, Linq or IDataReader should not change much about perf or memory consumption provided you don’t use Linq method that materialize the whole data (exhaust the stream) or a big part of it. That means you will need to avoid ToArray(), ToList() methods and other methods like this, and concentrate only on “streamed” methods (enumerations, skips, while, etc.).