Question

I am writing into two tables on SQL server row by row from C#.

My C# app is passing parameters into 2 stored procedures which are each inserting rows into tables.

Each time I call a stored procedure I open and then close the connection.

I need to write about 100m rows into the database.

Should I be closing and opening the connection every time I call the stored procedure?

Here is an example what I am doing:

public static void Insert_TestResults(TestResults testresults)
        {
            try
            {
                DbConnection cn = GetConnection2();
                cn.Open();

                // stored procedure
                DbCommand cmd = GetStoredProcCommand(cn, "Insert_TestResults");
                DbParameter param;

                param = CreateInParameter("TestName", DbType.String);
                param.Value = testresults.TestName;
                cmd.Parameters.Add(param);


                if (testresults.Result != -9999999999M)
                {
                    param = CreateInParameter("Result", DbType.Decimal);
                    param.Value = testresults.Result;
                    cmd.Parameters.Add(param);
                }


                param = CreateInParameter("NonNumericResult", DbType.String);
                param.Value = testresults.NonNumericResult;
                cmd.Parameters.Add(param);

                param = CreateInParameter("QuickLabDumpID", DbType.Int32);
                param.Value = testresults.QuickLabDumpID;
                cmd.Parameters.Add(param);
                // execute
                cmd.ExecuteNonQuery();

                if (cn.State == ConnectionState.Open)
                    cn.Close();

            }
            catch (Exception e)
            {

                throw e;
            }

        }

Here is the stored procedure on the server:

USE [SalesDWH]
GO
/****** Object:  StoredProcedure [dbo].[Insert_TestResults]    Script Date: 12/26/2011 10:45:08 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:      <Author,,Name>
-- Create date: <Create Date,,>
-- Description: <Description,,>
-- =============================================
ALTER PROCEDURE [dbo].[Insert_TestResults]
    -- Add the parameters for the stored procedure here

    @TestName varchar (500),
    @Result decimal (18,4)=null,
    @NonNumericResult varchar (50)=null, 
    @QuickLabDumpid int

AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

INSERT INTO [SalesDWH].[dbo].[TestResults]
           ([TestName]
           ,[Result]
           ,nonnumericresult
           ,[QuickLabDumpid])
     VALUES
           (@TestName,@Result,@nonnumericresult,@QuickLabDumpID)


END

For about 100m rows it will take like 3 days. This seems too slow for me. What can I do to speed this up? What are the standards on opening/closing connection so many times?

Answer 1

If you’re on SQL Server 2008 you could send multiple records at once through a table-valued parameter:

create type testResultUpload as table
(
    TestName varchar(500),
    Result decimal(18,4) null,
    NonNumericResult varchar(50) null, 
    QuickLabDumpid int
)

Then you could build up a DataTable on the client side and pass it to sql as one chunk. Though, you may want to do a thousand at a time to start off with.

You’d have to ammend your stored procedure to deal with an input record set, starting with the parameter definition

alter proc Insert_TestResult
(
    @testResultUpload testResultUpload readonly -- tvp must be readonly
)
as begin       

    -- This is short and sweet for demonstrative purposes
    -- but you should explicitly list your columns
    insert [SalesDWH].[dbo].[TestResults] 
    select
     *
    from @testResultImport

end

Then on your client side:

// create your datatable in the form of the newly created sql type
var dt = new DataTable();
dt.Columns.Add("TestName", typeof(String));
dt.Columns.Add("Result", typeof(Decimal));
dt.Columns.Add("NonNumericResult", typeof(String));
dt.Columns.Add("QuickLabDumpid", typeof(String));

// add your rows here (maybe do it in steps of a thousand
// 100 Million over the pipe at once is ill-advised)
// call the following code to hit sql

using (var cnx = new SqlConnection("your connection string"))
using (var cmd = new SqlCommand {
    Connection = cnx,
    CommandType = CommandType.StoredProcedure,
    CommandText = "dbo.Insert_TestResults",
    Parameters = {
        new SqlParameter {
            ParameterName = "@testResultUpload",
            Value = dt,
            SqlDbType = SqlDbType.Structured // make sure to specify structured
        }
    }
})
{
    cnx.Open();
    cmd.ExecuteNonQuery();
}