Question

I have an application that is made up of the following:

A central database containing 100k+ records A number of ‘client’ databases each containing around 10-20k records

The client databases contain details of contacts that each have a unique ID (contactID).
The central database contains some of these contacts identified by the same contactID.

Overnight we need to iterate through the client databases querying the central database for updates to each contact then bring it into the client database.
The central database is held by a third party so we cannot change anything.
The company holding the central database want to do this over web services iterating through each contact.

My concerns are this would be very slow to do over web services given the number of records.

Currently I am thinking of generating a file on each client which contains a list of all the contacts for that client. This would then be sent to the central database. The central database would then process this file and send back another file containing all updates.

How would you create this so it runs as quick as possible?

Answer 1

I would not go through each record in the scenario you describe. Web services are fine but you could as easily use them for bulk updates.

From the top of my head, something like this would work a little better:

1. Get a ‘diff‘ containing all the changes in the master database. This could be done before the synchronisation starts and only has to be done once per day.
2. Send a full list of contacts from the client to the server. With a list that big it does not really matter how it is send but if you don’t like SOAP web services think about JSON. I would go with web-services for convenience. Again, the list of contacts could be prepared in advance.
3. Create a list of records which need to be update by comparing the ‘diff‘with the list from the client.
4. Get all the updated details in one go (if you use MS SQL you could do that with XML, other SQL services offer different paths)
5. Send the details to the client

The same mechanism could be used for updating one record only, you just need to use different list (containing 1 ID) in the p.2