Question

I have a number of machines each with a Django instance, sharing a single Postgres database.

I want to run Celery, preferably using the Django broker and the Postgres database for simplicity. I do not have a high volume of tasks to run, so there is no need to use a different broker for that reason.

I want to run celery tasks which operate on local file storage. This means that I want the celery worker only to run tasks which are on the same machine that triggered the event.

Is this possible with the current setup? If not, how do to it? A local Redis instance for each machine?

Answer 1

I worked out how to make this work. No need for fancy routing or brokers.

I run each celeryd instance with a special queue named after the host. This can be done automatically, like:

./manage.py celeryd -Q celery,`hostname`

I then set up a hostname in the settings.py that stores the hostname:

import socket
CELERY_HOSTNAME = socket.gethostname()

In each Django instance this will have a different value.

I can then specify this queue when I asynchronously call my task:

my_task.apply_async(args=[one, two], queue=settings.CELERY_HOSTNAME)