Question

I need to update my contacts database in SQL Server with changes made in a remote database (also SQL Server, on a different server on the same local network). I can’t make any changes to the remote database, which is a commercial product. I’m connected to the remote database using a linked server. Both tables contain around 200K rows.

My logic at this point is very simple: [simplified pseudo-SQL follows]

/* Get IDs of new contacts into local temp table */

Select remote.ID into #NewContactIDs
From Remote.Contacts remote
Left Join Local.Contacts local on remote.ID=local.ID
Where local.ID is null

/* Get IDs of changed contacts */

Select remote.ID into #ChangedContactIDs
From Remote.Contacts remote
Join Local.Contacts local on remote.ID=local.ID
Where local.ModifyDate < remote.ModifyDate

/* Pull down all new or changed contacts */

Select ID, FirstName, LastName, Email, ...
Into #NewOrChangedContacts
From Remote.Contacts remote
Where remote.ID in (
        Select ID from #NewContactIDs 
        union 
        Select ID from #ChangedContactIDs
    )

Of course, doing those joins and comparisons over the wire is killing me. I’m sure there’s a better way – advice?

Answer 1

Consider maintaining a lastCompareTimestamp (the last time you did the compare) in your local system. Grab all the remote records with ModifyDates > lastCmpareTimestamp and throw them in a local temp table. Work with them locally from there.