Question

I want to create CLR UDF to scan a SQL table and read each column and sum all data for
each column (I am working with big datasets, like >1000 columns and 20,000,000>rows).

I’d like to compare applying for each column SUM(COLUMN_NAME) SQL function with parallelized for loop

So The SQL would look like

SELECT SUM(COLUMN_1),SUM(COLUMN_2),SUM(COLUMN_3),...,SUM(COLUMN_1000)

How can I do a CLR UDF in C# that would do that?

I am planning to use an array, so each time I read the row I add it like:
array[i]+= sqlValue;

How to do this, so I can execute both of them in a stored proc?

Table

column_1  column_2 column_3   .... column_1000
---------------------------------------------
     451      57     253                 135
     251      77     356                 965
     481      15     323                 655  
     452      15     135                 665
    ... 
    ...20,000,000 more rows

Answer 1

So what you’re trying to achieve is to do a thousand SELECT column_x FROM table, one for each column, and do the summing from hand.

That means 1000 parallel connections to the database all working within the same table within the same rows, locking each other (until you use with nolock).

The benefit to SELECT sum(column_1), sum(column_2), .... I can’t see one and I believe what you’re trying will be magnitudes slower than letting SQL Server doing what it does best.

EDIT:

As per your request here’s a quick’n’dirty sample, not tested since I currently have no SQL server at hand. I assumed the columns are of type long and the result is of type decimal.

public partial class UserDefinedFunctions
{
    private static string GetCommandText(int column)
    {
        return string.Format("select column_{0} from table", column);
    }

    [Microsoft.SqlServer.Server.SqlFunction(
        DataAccess = DataAccessKind.Read,
        TableDefinition = "result decimal",
        FillRowMethodName = "FillRow",
        SystemDataAccess = SystemDataAccessKind.Read)]
    public static IEnumerable fnSum(int columnNo)
    {
        var values = new List<long>();

        using (var cmd = new SqlCommand(GetCommandText(columnNo), new SqlConnection("context connection=true")))
        {
            cmd.Connection.Open();
            using (var reader = cmd.ExecuteReader(CommandBehavior.SingleResult | CommandBehavior.CloseConnection))
            {
                while (reader.Read())
                {
                    values.Add(reader.GetInt64(0));
                }
            }
        }

        return list;
    }

    private static void FillRow(object obj, out decimal result)
    {
        var values = (List<long>)obj;

        result = values.Sum(value => (decimal) value);
    }
}