Question

Through my ASP.NET / SQL Server 2008 app I need to do a bulk insert of records from a CSV file (maybe a million records). I will import them into a staging table first, so I can manipulate some of the data before moving it to a permanent table.

This will happen on a regular basis. And multiple imports may happen simultaneously. I also have to tell each import from the others.

My original plan was to use a column that had an Import_ID in it. But I see that Bulk Insert won’t allow me to set a field value.

Doing a search, I see that I can do a Bulk Insert into a view. And I’m guessing that the view can have a named parameter (Import_ID). But I haven’t really learned setting up parameters yet, so I don’t know if this is possible, or how to do it.

Can someone please tell me how to do this, or let me know another solution?

Thanks

Answer 1

You could bulk insert into a temporary staging table, for example since you know your SPID (and assuming you can trust the schema of some static table) you can say something like this, specifying the @filepath for the CSV file and the @ImportID, creating a table with your session id as a suffix, and doing all your work in a single dynamic SQL batch:

DECLARE @sql NVARCHAR(MAX), @spid VARCHAR(12) = RTRIM(@@SPID);

SET @sql = N'SELECT * INTO dbo.Stage' + @spid 
    + ' FROM dbo.RealStagingTable WHERE 1 = 0;';

SET @sql += N'BULK INSERT dbo.Stage' + @spid + ' FROM ('''
    + @filepath + ''' WITH (options);'

SET @sql += N'INSERT dbo.RealTable(ImportID, other columns)
  SELECT ' + RTRIM(@ImportID) + , other columns
  FROM dbo.Stage' + @spid + ';';

SET @sql += N'DROP TABLE dbo.Stage' + @spid + ';'

EXEC sp_executesql @sql;