We have an application that will be collecting data and storing it in local WinXP PCs using Microsoft SQL Server Compact. We want to aggregate that data up to a single full-blown SQL Server for reporting and archival. The data transport needs to be fairly continuous (i.e. not batched) though some latency is acceptable (a minute or two max).
Data is a one-way push from the collectors to the server. Collectors never need to know what other collectors are doing and the primary server will never be updating data back on the collector. Current plans are for 5 collectors, but it’s essentially unbounded for scalability.
We have to assume we’ll be ‘mostly connected’ but we cannot guarantee the connection from the collectors to the server. If the server or network go down, we’ll still be collecting and data will get pushed back up when the server is again reachable.
Ideally we’d like a solution that a non-programming engineer could set up once we’ve done the infrastructure work. So we’re fine writing some code and wizards, but the end user cannot be assumed to know anything about writing code, though they will have reasonable technical computer literacy.
Right now we have two technology candidates for this:
- SQL Replication
- Microsoft Sync Services
We have little experience with the first, but we know that setting up subscriptions, etc in SQL Server is painful and debugging them is not fun, so we’re trying to find an alternative.
We know almost nothing of #2, only that it’s been suggested as an alternative for getting device data to a server.
Does anyone have experience in this type of a scenario or with either/both of these technologies or anything we’re not thought of that they can share? SQL Compact on the collectors is a fixed requirement. SQL Server on the server is not required, but desired since the customer already has it.
I ended up going with option 3: neither. Instead we just periodically (user adjustable, but defaults to 5 seconds) use the SqlBulkCopy class to copy the records across. This works well becasue it allows us to pass in an IDataReader, so we locally open the table using TableDirect, seek to the highest RowID from the remote table, then pass the reader to the WriteToServer class.