Question

I have a few jobs that insert large data sets from a text file. The data is loaded via .NET’s SqlBulkCopy.

Currently, I load all the data into a temp table and then insert it into the production table. This was an improvement over straight importing into production. The T-SQL insert results query was a lot faster. Data is only loaded via this method, there is no other inserts or deletes.

However, I’m back to timeouts because of locks while the job is executing. The job consists of the following steps:

1. load data into temp table
2. start transaction
3. delete current and future dated rows
4. insert from temp table
5. commit

This happens once every hour. This portion takes 70 seconds. I need to get that to the smallest number possible.

The production table has about 20 million records and each import is about 70K rows. The table is not accessed at night, so I use this time to do all required maintenance (rebuild stats, index, etc.). Of the 70K, added, ~4K is kept from day-to-day – that is, the table grows by 4k a day.

I’m thinking a 2 part solution:

The job will turn into a copy/rename job. I insert all current data into the temp table, create stats & index, rename tables, drop old table.

Create a history table to break out older data. The “current” table would have a rolling 6 months data, about 990K records. This would make the delete/insert table smaller and [hopefully] more performant. I would prefer not to do this; the table is well designed with the perfect indexes; queries are plenty fast. But eventually it might be required.

Edit: Using Windows 2003, SQL Server 2008

Thoughts? Other suggestions?

Answer 1

Well one really sneaky way is to rename the current table as TableA and set up a second table with the same structure as TableB and the same data. Then set up a view with the same name and the exact fields in the TableA. Now all your existing code will use the view instead of the current table. The view starts out looking at TableA.

In your load process, load to TableB. Refresh the view defintion changing it to look at TableB. Your users are down for less than a second. Then load the same data to TableA and store which table you should start with somewhere in a database table. Next time load first to TableA and then change the view to point to TableA then reload TableB.