Question

I have a table with about 5 million records and I only need to move the last 1 million to production (as the other 4 million are there). What is the best way to do this so I don’t have to recopy the entire table each time?

Answer 1

A little faster will probably be:

Insert into prod.dbo.table (column1, column2....)
Select column1, column2.... from dev.dbo.table d
where not exists (
    select 1 from prod.dbo.table pc where pc.pkey = d.pkey
    )

But you need to tell us if these tables are on the same server or not

Also how frequently is this run and how robust does it need to be? There are alternative solutions depending on your requirements.

Given this late arriving gem from the OP: no need to compare as I know the IDs > X , then you do not have to do an expensive comparison. You can just use

Insert into prod.dbo.table (column1, column2....)
Select column1, column2.... from dev.dbo.table d
where ID > x

This will be far more efficient as you are only transferring the rows you need.