I developed a BizTalk application that receives as input a file that contains a bunch of messages. I use the BizTalk XML disassembler component to ‘debatch’ the file in sepereate messages. Each of those messages is picked up from the MessageBox by an orchestration that transforms the message and calls a wcf service.
The problem I am experiencing now is that each batch contains 1000 messages, and that those 1000 messages all seem to call the wcf service at once. The wcf service is “bombed” by those messages and is configured to process only 10 messages in parallel (each call has to process data and put data in a database) and returns a bunch of “Too Busy” exceptions back to BizTalk. I configured the wcf adapter to retry the connection again after 1 minute.
The end result is that BizTalk first debatches the messages, then bombs the wcf service with all 1000 messages, gets a bunch of “Too busy” exceptions, then waits while doing nothing, until 1 minute is passed, then bombs it again, and so on.
The processing would be much more efficient if I could configure BizTalk to open max 10 connections to that specific wcf service, but as far as I know, this is not possible. (The wcf service is configured to use net.tcp.)
I did already try the throttling settings of the host in several different ways, but either it is not helping, or it is making the application unbearable slow. Also the throttling in BizTalk seems to be implemented in a way that it first bombs a service, then notices that it was bombing, then waits a while doing nothing, and then lifts the throttle and start bombing again. It seems much better to trickle the requests/messages, so that they are much more evenly spread in time. I would like to configure the WCF adapter to receive max 4 messages per second for example. The throttling that is possible now says something like: over the sliding window of 5 seconds, i want throttling to be activated if there are more than 20 messages. But that is not the same, because it allows a “burst” effect.
Any ideas how i can improve throughput?
This question is already over a year old, but I just want to add an answer in case someone has the same problem.
I tried playing with the throttling configuration of the BizTalk host. This did not help. I did not actually try to use the singleton pattern, because that is something I do not want: we created a powerful service oriented architecture that can easily process several message in parallel, and I do not want to completely undo that by introducing a singleton pattern.
So what did I end up doing then? First I considered again what is actually required: we need to process a bunch of files that each contain 1000 messages. The order in which a message inside a file is processed, is not important. The order in which files are processed, is important. Normally, we should process first file 1, then 2, then 3, and so on. However, it is not that strict, the order is only on ranges of files, e.g., first range 1-5 has to be processed, then range 6-8, but within a range, the order of the files is not important. So those were the requirements.
The first thing that I changed, is instead of processing 1 message at a time, I change the service to accept a collection of messages, so that I can process 1 file at a time. By processing 1 file at a time, there is only 1 call to the WCF service, which has the advantage that there is a lot less chit-chat between BizTalk and the WCF service. Note however, that this makes the code on the WCF service side more complicated because each message still has to be processed independently of the others (makes error handling more complex). If we manage to process a limited number of files at once, we can also avoid the too busy error.
Next to the actual processing of messages, the WCF service also provides calls to “register” the processing of a file. This is code on the server side that checks whether a file can be processed at that time: it takes into account the order of the files and makes sure that a file (range) can only be registered when the previous file (range) is already processed. These register calls try to register a file (range) in a loop with a wait inside. The call tries to register the file, if it is not accepted, it waits and then tries again. I don’t really like this solution, but it works.
So in the end I have a solution that takes into account the order of file ranges, and next to that has a configuration on how many files can be processed in parallel. This means that I don’t get any too busy errors anymore. Not completely satisfied with my solution, but it does work and is very stable. It has been running without problems for the past year.