Question

We have built a C# .NET system that can be used to create data warehouses. This system takes selected databases and run a script against these databases to create a combined database/warehouse.

Now, I have three databases to be compiled into a single database and I am copying two tables from each (table [XI] and table [XII] – which have a one to many relationship, but have no constraints set up at the time of the copy/INSERT INTO). The figures for the script to run and the relevant sizes for each table are below:

The executed script consists of 30 SQL queries.

1. DatabaseA:

   Table [XI]  29,026 Rows (size 20,128Kb).
   Table [XII] 531,958 Rows (size 50,168Kb).
   Time taken for entire script: 1.51s.
   

2. DatabaseB:

   Table [XI]  117,877 Rows (size 17,000Kb).
   Table [XII] 4,000,443 Rows (size 512,824Kb).
   Time taken for entire script: 2.04s.
   

These both run fine and fast. The next is almost exactly the same size as the first but takes 40x as long!

1. DatabaseC:

   Table [XI]  29,543 Rows (size 20,880Kb).
   Table [XII] 538,302 Rows (size 68,000Kb).
   Time taken for entire script: 44.38s.
   

I cannot work out why this is taking so long. I have used SQL Server Profiler and the Performance Monitor, but I cannot nail-down the reason for this massive change in performance.

The query being used to do the update is dynamic and is shown at the bottom of this question – it is large due the explicit reference to the required columns. My question is; what could be causing this inordinate increase in execution time?

Any clues would be greatly appreciated.

SQL:

DECLARE @DbName NVARCHAR(128);
SET @DbName = (SELECT TOP 1 [DbName] 
               FROM [IPACostAdmin]..[TmpSpecialOptions]);
DECLARE @FilterSql NVARCHAR(MAX);
SET @FilterSql = (SELECT TOP 1 [AdditionalSQL] 
                  FROM [IPACostAdmin]..[TmpSpecialOptions]);
DECLARE @SQL NVARCHAR(MAX);
DECLARE @SQL1 NVARCHAR(MAX);
DECLARE @SQL2 NVARCHAR(MAX);
SET @SQL1 = 
    'INSERT INTO [' + @DbName + ']..[Episode] 
        ([Fields1], ..., [FieldN])'; 
SET @SQL2 = 
'SELECT 
     [Fields1], ..., [FieldN] 
FROM [B1A] ' + @FilterSql + ';'; 
SET @SQL = @SQL1 + @SQL2;
EXEC(@SQL);
GO

Note: I am splitting the dynamic SQL into @SQL1 and @SQL2 for clarity. Also note that I have not shown all columns due to space and the fact that it would largely be redundant.

Edit1.

1. The databases are on the same server.

2. The database files, including logs are in the same directory on the same drive.

3. There are no primary/foriegn keys or constraints set up on the source databases (DatabaseA/B/C) or the data warehouse database at the time of this INSERT INTO.

Edit2. I have ran the above query in management studio and it took 5s!?

Edit3. I have added a temporary CLUSTERED INDEX in a hope that this would assist this query, this has not helped either.

Answer 1

This was caused by a DELETE query being run before a preceeding CREATE CLUSTERED INDEX query had time to update the entire table. The solution was to use the BEGIN TRANSACTION and COMMIT keywords. This forces SQL Server to finish the indexing before attampting anyother operations.

Note, that this problem is only likely to arise when following a CREATE CLUSTERED INDEX query with a dynamic SQL statement that modifies the existing tabel.

I hope this helps someone else.