Question

I’m experiencing OutOfMemory exceptions in my application, when fetching from the database. It is a C# .Net application using Linq2Sql.

I have tried using GC.GetTotalMemory() to see how much memory is taken up before and after the call to the database. This gives a nice although not quite accurate picture of what is going on. When I look in the Windows Task Manager I can see that the Peak Working Set is not smaller when fetching the data in a paged manner using the following code:

public static void PreloadPaged()
{
    int NoPoints = PointRepository.Count();
    int pagesize = 50000;
    int fetchedRows = 0;

    while (fetchedRows < NoPoints)
    {
        PreloadPointEntity.Points.AddRange(PointRepository.ReadPaged(pagesize, fetchedRows));
        PointRepository.ReadPointCollections();
        PreloadPointEntity.PointCollections.Count());
        fetchedRows += pagesize;
    }
}


private static List<PointEntity> ReadPaged(int pagesize, int fetchedRows)
{
    DataModel dataContext = InstantiateDataModel();
    var Points = (from p in dataContext.PointDatas
                select p.ToEntity());

    return Points.Skip(fetchedRows).Take(pagesize).ToList();
}

I guess it’s the Linq2Sql code that is using up the memory and not reusing it or freeing it afterwards, but what can I do to get the memory foot print down?

I have observed that it uses 10 times as much memory to fetch the data as it does to store them in my list of enties. I have considered invoking the garbage collector, but I would rather avoid it.

Answer 1

you are retrieving way too much data and storing it in memory, that’s why you are getting an OOM exception.

1 of 2 things is occurring:

1. you are loading an excessive amount of data when the user will only view a subset of the results and/or this is a 1st attempt at “caching” data.
2. you do need all this data, but are using the wrong technology (Linq2Sql) to access the data.

if it’s the first, you need to either

1. load smaller chunks of data (20-50 records, not 50K or everything)
2. if this is only for display purposes, then query a projection of what’s needed, rather than the entity itself.

if it’s the second than use an ETL tool designed to manage large amounts of data. I prefer Rhino.ETL but SSIS also works.