Question

The work I do involves downloading into memory HUGE amounts of data from a SQL server database. To accomplish this, we have custom dataset definitions that we load using a SqlDataReader, then iterate through the Datatable and build each row into an object, and then usually package those objects into a massive dictionary.

The amount of data we are using is large enough that sometimes it cannot fit into a single datatable which have a memory cap. The dictionaries have even grown large enough to surpass 8 gb’s of system memory in the most extreme cases. I was giving the task of fixing the outofmemory exceptions being thrown when the datatables overflowed. I did this by implementing a batch process method that seemed to be in conflict with how datatables are meant to be used, but it worked for the time being.

I now have the task of further reducing the memory requirements of this process. My idea is to create a generically typed class inheriting from IEnumerator that takes a SqlDataReader and essentially uses the reader as the collection it is Enumerating. The MoveNext() function will advance the reader, and the Current property will return the typed object specified built from a builder method from the reader’s current row.

My question: Is this a feasible idea? I’ve never heard/can’t find online anything like it.

Also, logistically: How would I call the specific builder function that the type declaration demands when the Current property is called?

I’m open to criticism and chastising for dreaming up a silly idea. I’m most interested in finding the best practice for approaching the overall goal.

Answer 1

Seems reasonably sensible, and actually pretty straightforward using an iterator block:

private static IEnumerable<Foo> WrapReader(SqlDataReader reader)
{
    while (reader.Read())
    {
        Foo foo = ...; // TODO: Build a Foo from the reader
        yield return foo;
    }
}

Then you can use it with:

using (SqlDataReader reader = ...)
{
    foreach (Foo foo in WrapReader(reader))
    {
        ...
    }
}

You can even use LINQ to Objects if you’re careful:

using (SqlDataReader reader = ...)
{
    var query = from foo in WrapReader(reader)
                where foo.Price > 100
                select foo.Name;
    // Use the query...
}