Question

I have a scenario where data needs to be imported from a CSV file through a .NET application to SQL Server 2008 on a database server (distributed). The CSV file contains around 4 million ‘records’ while the destination table can contain over 100 million records. The data that is being imported needs to be checked against existing data for duplicates before the insert is done.

I tried building a DataTable in memory and sending this as a parameter to a stored procedure, however this is performing very badly.

What is a good approach for this scenario?

• Data for import (importdata) needs to be sent to SQL Server
• importdata needs to be checked agains existing data (existing) for duplicates
• if just a single duplicate is found in existing the entire import for importdata needs to be aborted

CSV Format

"Name1", "11111111-1111-1111-1111-111111111111"
"Name1", "11111111-1111-1111-1111-111111111111"
"Name1", "11111111-1111-1111-1111-111111111111"

Data for import (DataTable format schematically):

Table (
    name nvarchar(20),
    someId uniqueidentifier
)

Destination table on SQL Server (schematically):

Table (
    id int primarykey,
    name nvarchar(20),
    someId uniqueidentifier
)

Answer 1

What I would do . . .

1. Using the SqlBulkCopy class I’d insert the 4 million rows into the
   database into a temporary table (or a table that’s permanent but
   used only for this purpose).
2. Write a Proc that runs one table against the other, and only inserts
   rows to destination table appropriately.

The SqlBulkCopy class is by far the fastest way to get data into a MS SQL DB, and then get the db to do the update, it’s what it’s good at.

Update Can this be done in a transaction?

Yes, but I wouldn’t like to imagine a transaction covering the insertion of 4 million rows and updating a different 4 million rows.

If you use a truly temporary table (e.g. #MyTempStrore) then the temp table will be dropped when you finish that session on the database (e.g. dbConn.Close()).
If you use a permant table, you can create a unique session Id, insert all the rows with that Id and pass that Id to the stored proc. You can then clear those ids out of the table when you wish (or just truncate the table).

Hope this helps.