Question

We have a cloud based SaaS application and many of our customers (school systems) require that a backup of their data be stored on-site for them.

All of our application data is stored in a single MS SQL database. At the very top of the “hierarchy” we have an “Organization“. This organization represents a single customer in our system. Each organization has many child tables/objects/data. Each having FK relationships that ultimately end at “Organization”.

We need a way to extract a SINGLE customer’s data from the database and bundle it in some way so that it can be downloaded to the customers site. Preferably in a SQL Express, SQLite or an access database.

For example: Organization -> Skill Area -> Program -> Target -> Target Data are all tables in the system. Each one linking back to the parent by a FK. I need to get all the target data, targets, programs and skill areas per organization and export that data.

Does anyone have any suggestions about how to do this within SQL Server, a C# service, or a 3-rd party tool?

I need this solution to be easy to replicate for each customer who wants this feature “turned on”

Ideas?

Answer 1

I’m a big fan of using messaging to propagate data at the moment, so here’s a message based solution that will allow external customers to keep a local, in sync copy of the data which you provide on the web.

The basic architecture would be an online, password secured and user specific list of changes which have occurred in the system.
At the server side this list would be appended to any time there was a change to an entity which is relevant to the specific customer.
At the client would run an application which checks the list of changes for any it hasn’t yet received and then applies them to its local database (in the order they occurred).

There a a bunch of different ways of doing the list based component of the system but my gut feeling is that you would be best to use something like RSS to do this.

Below is a practical scenario of how this could work:

1. A new skill area is created for organisation “my org”
2. The skill is added to the central database and associated with the “my org” reccord
3. A SkillAreaExists event is also added at the same time to the “my org” RSS with JSON or XML data specifying the properties of the new skill area
4. A new program is added to the skill area that was just created
5. The program is added to the central database and associated with the skill area
6. A ProgramExists event is also added at the same time to the “my org” RSS with JSON or XML data specifying the properties of the new program
7. A SkillAreaHasProgram event is also added at the same time to the “my org” RSS with JSON or XML data specifying an identifier for the skill area and program
8. The client agent checks the RSS feed and sees the new messages and processes them in order
9. When the SkillAreaExists event is processed a new Skill area is added to the local DB
10. When the ProgramExists event is processed a new Program is added to the local DB
11. When the SkillAreaHasProgram event is processed the program is linked to the skill area

This approach has a whole bunch of benefits over traditional point in time replication.

• Its online, a consumer of this can get realtime updates if required
• Consistancy is maintained by order, at any point in time in the event stream if you stop receiving events you have a local DB which accuratly reflects the central DB as at some point in time.
• Its diff based, you only need to recieve changes
• Its auditable, you can see whats actually happened not just the current state.
• Its easily recoverable, if there’s a data consistency issue you can revert the entire DB by replaying the event stream.
• It allows for multiple consumers, lots of individual copies of the clients info can exist and function autonomously.

We have had a great deal of success with these techniques for replicating data between sites especially when they are only sometimes online.