Question

(The following question is a bit long so if you’re not interested in persistent cashing and notifications feel free to press the x-button in your browser!)

We’ve been tasked to implement a persistent data store that will keep a lot of numerical information.

Basically the use case is:

1. You’ll get feeds with data updates and insertions which need to be propagated in the data store.

   1.1. Frequency of updates currently ranges from 1 feed from 1 to 10 secs. But later on we may get more so scalability maters.

   1.2. Volumes. Each feed is going to be roughly 100K of rows (even if a value does not change it’ll still be in the feed)

2. The values need to be persisted. Once this happens then the C# server needs to be notified of the even. Ideally with an event. Because the feeds will contained data that have not changes ideally the server event should come along with the delta and not just OnStuffChanged.

Implementation

Because the system is using SQL server some non-technical people have signed off usage of sql server for the implementation. This kind of smelled bad to me! But in any case did some research and found about a Wrapper on the SqlDependency API.

I saw however that the appi comes with a luggage, in the form of a long list of constraints(http://msdn.microsoft.com/en-us/library/ms181122%28v=sql.105%29.aspx). I also saw that MS say that it’s not that efficient for sub second performance (which is not currently the case, but could be in the future).

Specifically for performance & reliability MS say

“Query notifications might not be the best choice for applications when notifications must be received with a sub-second response time, when the network infrastructure is not both fast and reliable, or when the volume of notifications is very high.”

and suggest alternatives to query notification such as:

1. After Update trigger(I’m not a big fan of triggers tho) on the monitored table whose action is to use the SQL Server service broker to send a message to the update needing notification.
2. Implementation of custom middleware that stores and notifies.

(guys I’m getting to the point thanks for being patient!)

So the point is that I really feel that none of these ideas other than the custom middleware is good.

Questions

• Has anyone had hands on experience with SqlDependency and Service Broker? Do you reckon I should be starting a crusade against mgmt in order to prevent a tech mistake from happening in the 1st place?
• I kind of think that perhaps I should be using something like redis/memcached/mongo for the data cache. They offer persistency. I’m sure I can figure out a way to bridge them with a relational DB and provide the changed/new numbers to the server for processing. Doesn’t this make more sense?
• Maybe I’m just over-engineering the thing. Should I instead be trying other suggestions(a similar stackoverflow question suggested “)?

Apologies for the lengthy post! Thank you in advance if you’ve bothered reading up to this point!

Answer 1

Don’t use SqlDependency, is not the right scenario for it. SqlDependency is to monitor data that changes seldom (eg. reference data). With SqlDependency you only get notified that data has changed, w/o knowing what the change consists of, you will have to query yourself.

Service Broker on its own would fit the bill much better, as a means to decouple the notification from the consumption. There are large scale deployments doing this already. With SQL Server 2012 you can do multicast.

But absolutely not by doing a trigger. Notification should be a first class citizen in your application, not an afterthought stiched to the data model in atrigger (ughhh). Have explicit procedures for notifying the subscribers and have your application explicitlty call them. Do not mix your data model and yoru messaging schemas,m you’ll just introduce unnecessary coupling.

IF you want to go NoSQL way, that is a whole different discussion. Redis is the usual choice. But don’t dismiss true pub/sub messaging products, like ZeroMQ.

BTW, these kind of crusades (pro or against, I don’t care…) are best fought with prototypes.