I have a Topic exchange from which I’d like to distribute messages to two queues on two servers part of a cluster, in order to reduce memory pressure on any particular server. My consumers are periodically slow, and I sometimes run into the high memory watermark.
The way I tried to resolve this is by routing messages using an intermediate direct exchange, with two queues bound to the exchange:
a (topic) -> a1 (direct) -> q1/q2 (bound to routing key "a")
But the messages were routed to both queues, as AMQP intends. Anyone has ideas? What I need is an exchange that routes to one and only one queue, even if the routing key matches many queues. I’d prefer not to change my routing keys, but that could be arranged.
I found Selective routing with RabbitMQ, which may mean I’ll need to implement my own routing logic. Hopefully, this already exists somewhere else.
You could perhaps use the Shovel plugin – http://www.rabbitmq.com/shovel.html – to move messages from your intermediate exchange to the two queues.
If you set up two shovels, both consuming from a single queue on the direct intermediate exchange, they should be able to fight over the messages coming in (I’m assuming that you don’t care too much if the two recipient queues don’t get the incoming messages in a strict round robin fashion). The shovels then each publish to one of the two end queues, and can send through the ACKs from the end consumer.