Question

I have a small utility script written as .Net/C# console app purely for the reliable SQL Server drivers, that twice a day essentially iterates over 70,000 records in SQL Server DB, does a bit of JSON wizardy with a Linq Projection into a JObject and sends the Json Object over HTTP to a NoSQL db.

var  command = sqlConnection.CreateCommand();
            command.CommandText = "SELECT * FROM MotherFckinBigTable";
            var reader = command.ExecuteReader();

            int loopCount = 0;
            while(reader.Read()) {
 // convert reader to JSON object
 // stringify json object
 // webclient uploaddata to couchdb

}
}

Since I never have to go back or skip, its a simple start to end iteration, I’ve not used paging here or Linq or LLBLGen etc.., but I was wondering is this severely frowned upon, if reviewed would a DBA have a heart attack? Or in this scenario is it acceptable? Bearing in mind this a utility script.

Answer 1

In almost every case you should aim to keep close your database connection as soon as possible. I wouldn’t say that there is too much wrong with your query (assuming that it is fit for purpose). However there is a problem the amount of processing that you’re doing whilst keeping the database connection open. You might want to consider refactoring your utility to keep the database connection lifetime to a minmum by sending the processing of your domain object to a queue or another thread.

You might want to use something like the following:

using (var  command = sqlConnection.CreateCommand())
{
    // Use specific column names and the table schema to improve query execution speed.
    // Use the NoLock if phantom reads are not going to be a business issue.
    command.CommandText = "SELECT [ColumnName], [ColumnName] FROM [Schema].[TableName] WITH (NOLOCK) "; 
    var reader = command.ExecuteReader();

    while(reader.Read()) {
        // Set properties on domain object (or use AutoMapper) - and don't forget to use a try / catch / finally if it is needed.
        // Asynchronously send the domain object onto queue or another thread which will convert to Json and send to CouchDb.
    }
}