We’ve built up an application infrastructure based on ActiveMQ.
We can send and receive messages just fine, and for the most part things are pretty fast and OK.
However, we’ve noticed that if we submit a batch of messages ‘at once’, say 5,000 messages – that ActiveMQ will get the messages to the 3rd party application on the other end pretty quickly, and that this application will process also pretty quickly, and that it will en queue the replies back to the broker quickly also, say under a minute.
But for some reason, our VB.NET EXE that originated the messages in the first place only appears to be processing the return messages it receives erratically, sometimes doing about one per second, sometimes taking breaks for an hour or so and then going back to one per second.
Origin (VB.NET EXE which we manage) -> Broker (which we manage) -> (3rd party app) -> back to the same broker -> back to the origin app. The Receiver is waiting for the event MessageListener from C# code downloaded from ActiveMQ maybe 9 months ago:
Public Delegate Sub MessageListener(ByVal message As NMS.IMessage) Member of: NMS I think what’s happening is that MessageListener only gives us one message (NMS.IMessage) to chew on, and so that’s what we process.
Is there some way to say ‘On a MessageListener event, please see if there are other messages on the queue right now and do them all’?
Turns out, we think we know a bit more now what this is about.
When our VB.NET WinForms app that uses the ActiveMQ DLL eventually crashes, which it tends to do a few times a week, we have a watchdog program that uses the Winternals pslist and pskill utilities to reap the zombie, and then start a new client connection.
When this happens, using jconsole to analyze the broker shows us that the zombie’s session is still registered, and so is the fresh new client.
My theory right now is that when AMQ sees both sessions, it tries to start distributing messages to both sessions round-robin style. AMQ tries to send the message to the zombie, which does not respond. After a certain amount of time (one second perhaps) AMQ gives up and goes to the next session in the list, the new fresh client.
At some point, the broker or TCP stack probably notices that the zombie has not kept it’s TCP connection active and it gives up; then operation goes back to normal.
So the question becomes, how to write an ActiveMQ client that a) does not die or b) dies gracefully, shutting down it’s session in the process?
Edit: upgrading to the next version of ActiveMQ solved this. Also we had a single app doing the sending and receiving, but it was not threadsafe – so if it received while it was trying to send this caused the crashes. We re-wrote it as two console apps, one that sent data and one that received data. No more crashes. Also the older version of ActiveMQ we were using at the time didn’t handle crashes gracefully, upgrading to 4.x solved that.