Question

Hi all i have the following merge sql script which works fine for a relatively small number of rows (up to about 20,000 i’ve found). However sometimes the data i have in Table B can be up to 100,000 rows and trying to merge this with Table A (which is currently at 60 million rows). This takes quite a while to process, which is understandable as it has to merge 100,000 with 60 million existing records!

I was just wondering if there was a better way to do this. Or is it possible to have some sort of count, so merge 20,000 rows from Table B to Table A. Then delete those merged rows from table B. Then do the next 20,000 rows and so on, until Table B has no rows left?

Script:

MERGE
    Table A AS [target]
USING
    Table B AS [source]
ON
    ([target].recordID = [source].recordID)
WHEN NOT MATCHED BY TARGET
    THEN
        INSERT([recordID],[Field 1]),[Field 2],[Field 3],[Field 4],[Field 5])
        VALUES([source].[recordID],[source].[Field 1],[source].[Field 2],[source].[Field 3],[source].[Field 4],[source].[Field 5]
    );

Answer 1

MERGE is overkill for this since all you want is to INSERT missing values.

Try:

INSERT INTO Table_A
([recordID],[Field 1]),[Field 2],[Field 3],[Field 4],[Field 5])
SELECT  B.[recordID],
        B.[Field 1],B.[Field 2],B.[Field 3],B.[Field 4],B.[Field 5]
FROM Table_B as B
WHERE NOT EXISTS (SELECT 1 FROM Table_A A
                  WHERE A.RecordID = B.RecordID)

In my experience MERGE can perform worse for simple operations like this. I try to reserve it for when you need varying operations depending on conditions, like an UPSERT.