Question

In our project, we want to use the RabbitMQ in “Task Queues” pattern to pass data.

On the producer side, we build a few TCP server(in node.js) to recv
high concurrent data and send it to MQ without doing anything.

On the consumer side, we use JAVA client to get the task data from
MQ, handle it and then ack.

So the question is:
To get the maximum message passing throughput/performance( For example, 400,000 msg/second) , How many queues is best? Does that more queue means better throughput/performance? And is there anything else should I notice?
Any known best practices guide for using RabbitMQ in such scenario?

Any comments are highly appreciated!!

Answer 1

For best performance in RabbitMQ, follow the advice of its creators. From the RabbitMQ blog:

RabbitMQ’s queues are fastest when they’re empty. When a queue is
empty, and it has consumers ready to receive messages, then as soon as
a message is received by the queue, it goes straight out to the
consumer. In the case of a persistent message in a durable queue, yes,
it will also go to disk, but that’s done in an asynchronous manner and
is buffered heavily. The main point is that very little book-keeping
needs to be done, very few data structures are modified, and very
little additional memory needs allocating.

If you really want to dig deep into the performance of RabbitMQ queues, this other blog entry of theirs goes into the data much further.