Question

we use celery with rabbitMQ backend and some of our servers hang with error: “[Errno 113] No route to host”(which can be a result of half of our servers being in US and half in Europe).

I need to be sure that every task is being delivered, unfortunately I have no idea how to retry tasks sent using send_task/string identifier(server that sends tasks has no access to code of remote worker) like this:

send_task("remote1.tasks.add_data", args=[...], kwargs={}, queue="remote1")

Is it possible to retry such task?

Answer 1

sent_task send just the message to the broker, if the exception is raised on the servers that call the sent_task, probably the message simply doesn’t reach the broker, than there is no task to retry but just an exception to be handled.
Otherwise if all you workers randomly raise this exception because they can’t reach the broker for some reason probably you can solve by set to true the celery conf vars

CELERY_ACKS_LATE = True

“Late ack means the task messages will be acknowledged after the task has been executed, not just before, which is the default behavior.”
This means that if something go mad during the execution of the task in the worker, the broker doesn’t receive the acks and another worker will execute the task.