Question

A third party application creates one database per project. All the databases have the same tables and structure. New projects may be added at anytime so I can’t use any EF schema.

What I do now is:

    private IEnumerable<Respondent> getListRespondentWithStatuts(string db)
    {
        return query("select * from " + db + ".dbo.respondent");
    }

    private List<Respondent> query(string sqlQuery)
    {
        using (var sqlConx = new SqlConnection(Settings.Default.ConnectionString))
        {
            sqlConx.Open();
            var cmd = new SqlCommand(sqlQuery, sqlConx);
            return transformReaderIntoRespondentList(cmd.ExecuteReader());
        }
    }

    private List<Respondent> transformReaderIntoRespondentList(SqlDataReader sqlDataReader)
    {
        var listeDesRépondants = new List<Respondent>();

        while (sqlDataReader.Read())
        {
            var respondent = new Respondent
            {
                CodeRépondant = (string)sqlDataReader["ResRespondent"],
                IsActive = (bool?)sqlDataReader["ResActive"],
                CodeRésultat = (string)sqlDataReader["ResCodeResult"],
                Téléphone = (string)sqlDataReader["Resphone"],
                IsUnContactFinal = (bool?)sqlDataReader["ResCompleted"]
            };

            listeDesRépondants.Add(respondent);
        }

        return listeDesRépondants;
    }

This works fine, but it is deadly slow (20 000 records per minutes). Do you have any hints on what strategy should be faster? For info, what is really slow is transformReaderIntoRespondentList method

Thanks!!

Answer 1

Generally speaking anything SELECT * FROM is bad practice, but it could also be resulting in you having to pull back more data than is actually required. The transform is operating on only a few columns are more columns than required being returned? Consider replacing with:

  private IEnumerable<Respondent> getListRespondentWithStatuts(string db)
  {
    return query("select ResRespondent, ResActive, ResCodeResult, Resphone, ResCompleted  from " + db + ".dbo.respondent");
  }

Also, gaurd against SQL-Injection attacks; concating strings for SQL queries is very dangerous.

When pulling data from a DataReader, I find that using the non-named lookups work best:

  var respondent = new Respondent
  {
    CodeRépondant = sqlDataReader.GetString(0),
    IsActive = sqlDataReader.IsDBNull(1) ? (Boolean?)null : sqlDataReader.GetBoolean(1),
    CodeRésultat = sqlDataReader.GetString(2),
    Téléphone = sqlDataReader.GetString(3),
    IsUnContactFinal = sqlDataReader.IsDBNull(4) ? (Boolean?)null : sqlDataReader.GetBoolean(4)
  };

I have not explcicitly tested the performance difference in a long while; but that used to make a notable difference. The ordinal checks did not have to do a named lookup and also avoided boxing/unboxing values.

Other than that, without more info it is hard to say… do you need all 20,000 records?

UPDATE

Ran a simple local test case with 300,000 records and reduced the time to load all data by almost 50%. I imagine these results will vary depending on the type of data being retrieved; but it still does make a difference on overall execution time. That being said, in my environment we are talking a drop from 650ms to just over 300ms.

NOTE

If respondent is a view, what is likely “really slow” is the database building up the result set; although the data reader will start processing information as soon as records are available, the ultimate bottleneck will be the database itself and/or network latency. Other than the above optimizations, there is not going to be much that you can do with your code unless you can index the view/table to optimize the query and or reduce the information required.