Question

I’m hoping to speed up some large scale ETL operations, that read and transform a bunch of stuff from strangely formatted flat files and an even stranger XML datastore, then insert many records into an Oracle dbms. I’m using ODP.NET 11g in C# / dotnet 4.

The OracleBulkCopy class seems like a good thing to try. The thing is, it wants to read its data from an IDataReader instance or an array of IDataRecord instances ( i.e. a resultset of some other dbms query).

Is there a straightforward way to wrap my data rows in an IDataReader or an array of IDataRecord classes, in memory so I can push them into OracleBulkCopy?

The examples I’ve found show data migration from one dbms to another. But, I’m trying to avoid writing these items one-by-one to a dbms so I can then bulk load them. I’d rather use the IDataReader equivalent of a memory stream. Thanks.

Answer 1

I had a similar problem, I was using Link2Sql and found problems in the performance. I have found a solution using reflection and extension methods, where I read the column data out of the ColumnAttribute.

public static void Insert<T>(this OracleBulkCopy copy, IEnumerable<T> entities)
    where T : class
{
    var properties = typeof(T).GetProperties();
    var columnNames = new List<string>();
    var columnPropertyNames = new List<string>();
    foreach (var propertyInfo in properties)
    {
        var attribute = Attribute.GetCustomAttribute(propertyInfo, typeof(ColumnAttribute)) as ColumnAttribute;
        if (attribute != null)
        {
            columnNames.Add(attribute.Name);
            columnPropertyNames.Add(propertyInfo.Name);
        }
    }

    copy.ColumnMappings.Clear();
    foreach (var columnName in columnNames)
    {
        copy.ColumnMappings.Add(columnName, columnName);
    }

    var table = new DataTable(copy.DestinationTableName);

    foreach (var columnMapping in copy.ColumnMappings)
    {
        table.Columns.Add(((OracleBulkCopyColumnMapping)columnMapping).DestinationColumn);
    }

    foreach (var entity in entities)
    {
        var row = table.NewRow();
        for (var i = 0; i < columnNames.Count; i++)
        {
            var value = typeof (T).GetProperty(columnPropertyNames[i]).GetValue(entity, null);
            if (value is DateTime)
                value = new OracleDate((DateTime) value);
            row[columnNames[i]] = value;
        }

        table.Rows.Add(row);
    }

    copy.WriteToServer(table);
}

The Entity looks like:

public class SomeEntity
{
  [Column(Name = "Id")]
  public string Id { get; set; }
  //more columns
}

Using it is as easy as:

using (var connection = CreateDbConnection())
{
    connection.Open();
    using (var copy = new OracleBulkCopy(connection))
    {
        copy.DestinationTableName = tableName;
        copy.BulkCopyTimeout = DefaultTimeoutInSeconds;
        copy.BatchSize = BatchSize;
        copy.Insert(entities);
    }
}