Question

I have a big table in SQL Server 2008 R2. It contains billions of rows. I need to load the whole data set in our application. Query the whole table is very slow. I want to use bcp dump it into a file and load it. But the problem is there are string columns it contains all kinds of special characters like ‘\t’, ‘\0’, comma, and ‘\n’. I can’t find a good field/row terminator. But long string terminator slows down the data file loading for my application. The question is:

1. Is there any API that load data faster then SQL query? I find there is a native importing API IRowsetFastLoad. But not lucky on exporting.
2. Is there any API for BCP native format? I can’t find any document about the format of native bcp file.

Answer 1

From BOL:

-n

Performs the bulk copy operation using the native (database) data types of the data. This option does not prompt for each field; it uses the native values.

Billions of rows? Then you will also want to use :

-b batch_size

Specifies the number of rows per batch of data copied. Each batch is copied to the server as one transaction. SQL Server commits or rolls back, in the case of failure, the transaction for every batch.

Can’t you access the two databases at once, perhaps through Linked Server? It would make things easier.

DECLARE @StartId BIGINT
DECLARE @NmbrOfRecords BIGINT
DECLARE @RowCount BIGINT

SET @StartId = 0
SET @NmbrOfRecords = 9999
SET @RowCount = 1

WHILE @RowCount > 0
BEGIN
  BEGIN TRANSACTION

  INSERT INTO DestinationDatabase.dbo.Mytable
  SELECT * FROM SourceDatabase.dbo.Mytable
  WHERE ID BETWEEN @StartId AND @StartId + @NmbrOfRecords

  SET @RowCount = @@ROWCOUNT

  SET @StartId = @StartId + @NmbrOfRecords + 1

  COMMIT TRANSACTION
END