Question

I have some tables in a MySQL database to represent records from a sensor. One of the features of the system I’m developing is to display this records from the database to the web user, so I used ADO.NET Entity Data Model to create an ORM, used Linq to SQL to get the data from the database, and stored them in a ViewModel I designed, so I can display it using MVCContrib Grid Helper:

public IQueryable<TrendSignalRecord> GetTrends()
{
    var dataContext = new SmgerEntities();
    var trendSignalRecords = from e in dataContext.TrendSignalRecords
                select e;

    return trendSignalRecords;
}

public IQueryable<TrendRecordViewModel> GetTrendsProjected()
{
    var projectedTrendRecords = from t in GetTrends()
    select new TrendRecordViewModel
    {
        TrendID = t.ID,
    TrendName = t.TrendSignalSetting.Name,
    GeneratingUnitID = t.TrendSignalSetting.TrendSetting.GeneratingUnit_ID,
    //{...}
    Unit = t.TrendSignalSetting.Unit
    };
    return projectedTrendRecords;
}

I call the GetTrendsProjectedMethod and then I use Linq to SQL to select only the records I want. It is working fine in my developing scenario, but when I test it in a real scenario, where the number of records is way greater (something around a million records), it stops working.

I put some debug messages to test it, and everything works fine, but when it reaches the return View() statement, it simply stops, throwing me a MySQLException: Timeout expired. That let me wondering if the data I sent to the page is retrieved by the page itself (it only search for the displayed items in the database when the page itself needs it, or something like that).

All of my other pages use the same set of tools: MVCContrib Grid Helper, ADO.NET, Linq to SQL, MySQL, and everything else works alright.

Answer 1

You absolutely should paginate your data set before executing your query if you have millions of records. This could be done using the .Skip and .Take extension methods. And those should be called before running any query against your database.

Trying to fetch millions of records from a database without pagination would very likely cause a timeout at best.