Question

I have to read 100.000 rows from database, I use this code :

List<elementClass> Listelement = new List<elementClass>();
using (DbDataReader reader = cmd.ExecuteReader())
{
    while (reader .Read())
    {
        elementClass element= new elementClass();
        DatabaseClass.ReadFromObject(reader , element);
        Listelement.Add(element);
    }
}

ReadFromObject :

public static void ReadFromObject(DbDataReader dr, Object obj)
{
    DataTable tableDescriptor = dr.GetSchemaTable();

    try
    {
        //Id - Identifier
        obj.GetType().GetProperty("Id").SetValue(obj, dr.GetValue(dr.GetOrdinal("Id")), null);

        PropertyInfo[] properties = obj.GetType().GetProperties();
        foreach (PropertyInfo pi in properties)
        {
            if (!ExistColumn(tableDescriptor, pi.Name)) continue;   // field not found
            if (dr[pi.Name] == DBNull.Value) continue;

            if (pi.PropertyType.IsEnum) { }
            else
            {
                switch (pi.PropertyType.Name)
                {
                    case "DateTime":
                        object dateValue = null;
                        if (dr[pi.Name].GetType().Name == "String")
                            dateValue = F24Common.Utility.DateFromString(dr.GetString(dr.GetOrdinal(pi.Name)));
                        else
                            dateValue = dr.GetDateTime(dr.GetOrdinal(pi.Name));
                        pi.SetValue(obj, dateValue, null);
                        break;
                    default:
                         pi.SetValue(obj, dr.GetValue(dr.GetOrdinal(pi.Name)), null);
                        break;
                }
            }
        }

    }
    catch (Exception )
    {

    }
}

There is something definitely wrong with this code, it’s too slow (the whole processing takes 45-50 minutes).
Is it possibile to fill the list using more than one CPU core (I have a quad-core CPU available) at the same time ?

Answer 1

At a glance, I would think that the bottleneck lies with the amount of reflection. I’d suggest having a read of Jon Skeets excellent Making Reflection fly article. You can then cache the appropriate delegates keyed on the type of obj.

I would be hesitant to access the DataReader from multiple threads as I would doubt that it is safe to do so.