Question

Is there a message queue implementation that allows breaking up work into ‘batches’ by inserting ‘message barriers’ into the message stream? Let me clarify. No messages after a message barrier should be delivered to any consumers of the queue, until all messages before the barrier are consumed. Sort of like a synchronization point. I’d also prefer if all consumers received notification when they reached a barrier.

Anything like this out there?

Answer 1

I am not aware of existing, widely-available implementations, but if you’ll allow me I’d propose a very simple, generic implementation using a proxy, where:

• producers write to the proxy queue/topic
• the proxy forwards to the original queue/topic until a barrier message is read by the proxy, at which point:
  • the proxy may notify topic subscribers of the barrier by forwarding the barrier message to the original topic, or
  • the proxy may notify queue subscribers of the barrier by:
    • periodically publishing barrier messages until the barrier has been cleared; this does not guarantee that all consumers will receive exactly one notification, although all will eventually clear the barrier (some may receive 0 notifications, others >1 notifications — all depending on the type of scheduler used to distribute messages to consumers e.g. if non-roundrobin)
    • using a dedicated topic to notify each consumer exactly once per barrier
  • the proxy stops forwarding any messages from the proxy queue until the barrier has been cleared, that is, until the original queue has emptied and/or all consumers have acknowledged all queue/topic messages (if any) leading up to the barrier
• the proxy resumes forwarding

UPDATE

Thanking Miklos for pointing out that under JMS the framework does not provide acknowledgements for asynchronous deliveries (what is referred to as "acknowledgements" in JMS are purely a consumer side concept and are not proxiable as-such.)

So, under JMS, the existing implementation (to be adapted for barriers) may already provide application-level acknowledgements via an "acknowledgement queue" (as opposed to the original queue — which would be a "request queue".) The consumers would have to acknowledge execution of requests by sending acknowledgement messages to the proxy acknowledgement queue; the proxy would use the acknowledgement messages to track when the barrier has been cleared, after having also forwarded the acknowledgement messages to the producer.

If the existing implementation (to be adapted for barriers) does not already provide application-level acknowledgements via an "acknowledgement queue", then you could either:

• have the proxy use the QueueBrowser, provided that:
  1. you are dealing with queueus not events, that
  2. you want to synchronize on delivery not acknowledgement of execution, and
  3. it is OK to synchronize on first delivery, even if the request was actually aborted and has to be re-delivered (even after the barrier has been cleared.) I think Miklos already pointed this problem out IIRC.
• otherwise, add an acknowledgment queue consumed by the proxy, and adapt the consumers to write acknowledgements to it (essentially the JMS scenario above, except it is not necessary for the proxy to forward acknowledgement messages to the producer unless your producer needs the functionality.)