Question

I’d like to create a farm of processes that are able to OCR text.
I’ve thought about using a single queue of messages which is read by multiple OCR processes.

I would like to ensure that:

• each message in queue is eventually processed
• the work is more or less equally distributed
• an image will be parsed only by one OCR process
• An OCR process won’t get multiple messages at once (so that any other free OCR process can handle the message).

Is that possible to do using AMQP?

I’m planning to use python and rabbitmq

Answer 1

Yes, that’s possible. Server cluster for a real-time MMO game I’m working on operate this way. We use ActiveMQ, but I think all this possible with RabbitMQ as well.

All items that you mentioned you get out of the box, except last one.

• each message in queue is eventually processed – this is one of main responsibilities of message brokers
• the work is more or less equally distributed – this is another one 🙂
• an image will be parsed only by one OCR process – the distinction of /topic and /queue exists for this. Topics are like broadcast signals, queues are tasks. You need a /queue in your scenario

To make last one work in desired way, consumers send AMQ-specific argument when subscribing to the queue:

activemq.prefetchSize: 1

This setting guarantees that consumer will not take any more messages after it took one and until it send an ack to AMQ. I believe something similar exists in RabbitMQ.