We are looking at integrating our platform with Salesforce via Salesforce Outbound Messaging (SOM). Every time client updates an object(s) in Salesforce, SOM will call our Webservice endpoint with the updated object(s) (up to 100 objects in a single call). Our Webservice needs to update corresponding records in our database.
SOM works quite well for our purposes except for 1 issue.
Some clients do mass nightly updates. It is not unusual for 200,000-500,000 objects to be updated. This means we will get 2,000-5,000 calls with 100 objects in very short time period. Our Webservice will be overwhelmed with so much data specially if multiple clients do mass updates close to each other.
To deal with this large volume/spike Webserver will create message on Application server for each object in the SOM call. Another process will take messages from Message Queue and update the database.
MSMQ is only limited by hardware so should be able handle millions of messages whilst we clear backlog.
The main question is this good design for dealing large amount of data/webservice calls? Is there better method?
If you’re concerned about your system’s ability to handle a large volume from salesforce in a short period of time, perhaps you should look at the replication api. It’s more of a pull model. You call salesforce when you’re ready to consume more data.
Editing to add that if storing a message in a queue is significantly cheaper than doing the ultimate processing of the message (which seems to be the case here), using a message queue seems like a good plan. I’m only nominally familiar with MSMQ. But assuming it’s remotely as enterprise grade as many of the free JMS queues, it should be up to the task.