Question

I have a logger in Celery (with RabbitMQ) and want to duplicate its work for emergency cases.

# tasks.py
@task
def log(message):
    with open('test.txt', 'a') as f:
        f.write(message)


# views.py
log.delay(message)

How can I make 2 instances of Celery on different machines run log() when it’s called?

Does it make sense at all to do this?

This is possible in RabbitMQ. If you have a topic-based exchange, it’s clear that one message can be put into two different queues and delivered independently to 2 receivers.

sender =>
[message, routing_key=event.logging.log] => [queue A, topic=event.#]     
                                                      => receiver 1
                                         => [queue B, topic=*.logging.*]
                                                      => receiver 2

The message will be sent to both queues, and none of them will steal the message from another one.

Answer 1

To do this you would have to configure the exchange to be a topic exchange (as you say):

CELERY_QUEUES = {
   'celery': {
       'exchange': 'celerytopic',
       'exchange_type': 'topic',
       'routing_key': 'celery',
   },
}

Then you can create your backup exchange using the AMQP api:

 from celery import current_app as celery

 with celery.broker_connection() as conn:
     conn.default_channel.queue_declare(queue='celery.backup', durable=True)
     conn.default_channel.queue_bind(queue='celery.backup',
                                     exchange='celerytopic',
                                     routing_key='celery',
                                     durable=True)

Since you already have a queue named celery you may have to delete that first:

$ camqadm queue.delete celery