Question

We have a singleton (sequential convoy) orchestration instance which has been flooded with about 14000 messages. This has meant that the instance has been processing for 8 days, but the rate of processing is currently very slow (about 1 per 7 minutes). The instance started fast, but has slowed down to this current performance level over the past few days.

We have diverted further messages away from this instance and our plan is to terminate it once it has finished processing through the backlog.

The problem is, based on the current processing rate we have calculated we have three days before it finishes (about 660 messages left to process).

My question is: Is there any way we can speed this up?

Answer 1

We managed to create a workaround to this:

1. Query the messagebox and retrieve all the messages which were consumed but not processed
2. Kill the slow-performing orchestration instance
3. Pipe all the unprocessed messages back in which created a new instance of the orchestration.

This new instance was unencumbered by all the history (consumed messages) of the previous instance, and so ran about 60 times faster, clearing the remaining backlog fast.

The way we identified consumed-but-not-processed messages was by using the method I documented here: Technique for knowing the number of consumed but unprocessed messages for a given BizTalk sequential convoy orchestration instance

The rest, once you have retrieved a list of message guids from the messagebox db is documented at the following locations:

• How to retrieve Message Body from BizTalk Database
• 3 ways of programmatically extracting a message body from the BizTalk tracking database (I used the 3rd one)

In summary once you have the message IDs you select the imgPart column from the Parts table in the messagebox db and this gives you an binary encoded, compressed version of the message body. You then use the code from the above articles to reconstruct the message.

After this it was just a case of getting all the messages and then firing them back into biztalk via a MSMQ receive location.

Longer term we will need to address the flaws in our design which led to this problem occuring – although flooding was not expected, to have the performance of a running process collapse through the floor under heavy load is never a good place to be.

The problem we have is that for the majority of our consumer systems, source message ordering must be maintained. For this reason we have singletons everywhere, all of which are potentially exposed to this problem.

I have posted about message re-sequencing strategies in BizTalk here: Resequencing strategies for ordered delivery in BizTalk