Question

I have a Qt application, that reads a special text file, parses it and inserts about 100000 rows into a temporary table in a firebird database. Then it starts a stored procedure to process this temporary table and apply some changes to permanent tables. Inserting 100000 rows into in-memory temporary table takes about 8 seconds on firebird.

Now I need to implement such behavior using MS SQL Server 2008. If I use simple serial inserts it takes about 76 seconds for 100000 rows. Unfortunately, it’s too slow. I looked at the following ways:

1. Temporary tables (# and ##). Stored on the disk in tempdb scheme. So there is no speed increase.
2. Bulk Insert. Very nice insertion speed, but thre is a need to have client or server-side shared folder.
3. Table variables. MSDN says: “Do not use table variables to store large amounts of data (more than 100 rows).”

So, tell me please, what is the right way to increse insertion speed from client application to MSSSQL2008.

Thank you.

Answer 1

You can use the bulk copy operations available through OLE DB or ODBC interfaces.

This MSDN article seems to hold your hand through the process, for ODBC:

1. Allocate an environment handle and a connection handle.

2. Set SQL_COPT_SS_BCP and SQL_BCP_ON to enable bulk copy operations.

3. Connect to SQL Server.

4. Call bcp_init to set the following information:

   • The name of the table or view to bulk copy from or to.

   • Specify NULL for the name of the data file.

   • The name of an data file to receive any bulk copy error messages
     (specify NULL if you do not want a message file).

   • The direction of the copy: DB_IN from the application to the view or
     table or DB_OUT to the application from the table or view.

5. Call bcp_bind for each column in the bulk copy to bind the column to a
   program variable.

6. Fill the program variables with data, and call bcp_sendrow to send a
   row of data.

7. After several rows have been sent, call bcp_batch to checkpoint the
   rows already sent. It is good practice to call bcp_batch at least once
   per 1000 rows.

8. After all rows have been sent, call bcp_done to complete the
   operation.

If you need a cross platform implementation of the bulk copy functions, take a look at FreeTDS.