I have a web service that runs 24/7. Periodically it needs to refresh its database with data from another web service. There is a lot of data. It’s tens of thousands of rows. (no, I don’t mean this is a lot of data for SQL Server, just trying to point out that I expect it to take some time to come down the pipe from the other web service)
The data refresh can take between 5 and 10 minutes. The actual data update portion of that is between 1 and 2 minutes. This means the service would be down for all intents and purposes when consumers would be requesting this type of data.
I would like to implement a system where data is always available. The only thing that comes to mind is some type of system where I maintain two separate databases. I populate the inactive one, swapping it to active before populating the other one.
I’m not sure I know the best way to do this. My current ideas just revolve around two sets of the schema in a single database (using views to access the active set) or two databases each with the same schema. The application would rotate between the two databases.
Any suggestions from someone who has done something like this before?
Dynamic sql! Generate random table name bulk the data into new table and create/replace the existing view with new table name/s. Or lookup table partitioning and dynamically handle it that way! (more elegant) – On off peak, do maintenance to consolidate your table from random tables.