Question

I would like to set up a simple python app as a periodic celery task on an Ubuntu instance. I have followed the Celery documentation to set up the structure as recommended:

proj/__init__.py
/celery.py
/tasks.py

This works fine and I can run the periodic task from ../proj using:

celery worker -B --app=proj

I then copied in the celeryd and celerybeat files and edited the default files.

When I start the daemon, however, celeryd runs but does not seem to find the celery.py config and logs a connection refused error.

I think it must be related to the paths in:

#/etc/default/celeryd:

# Where to chdir at start.
CELERYD_CHDIR="/home/user/"

CELERY_CONFIG_MODULE="/home/user/proj"

I have left the other settings as specified in the docs, apart from changing the CELERYD_USER.

I feel I should be using –app in the config as well and perhaps specifying my python environment but I’m not sure where or how (I’m not running a virtual env).

Answer 1

I am struggling with the same thing right now. This is how I see it:

There are two ends of the communication application that executes tasks and worker that executes task.

Worker does not need to know app! For the worker part it is just enough to know celeryconfig. You could pass it with celery worker –config=mypackage.celeryconfig. The worker will connect to the queue with BROKER_URL. Remember to declare there CELERY_IMPORTS there so worker would know where to look for task definitions.

The client app must know where to send it’s requests. So the same configuration must be passed with one of the methods of passing configuration file. I picked this one:

from __future__ import absolute_import

from celery import Celery
celery = Celery()
import backend.async.celeryconfig
celery.config_from_object(backend.async.celeryconfig)

This setup works for me when I run it without deamon, but for some reason deamon ignores my CELERY_CONFIG_MODULE settings.

---

Update:

CELERY_CONFIG_MODULE is not used in /etc/init.d/celeryd script anywhere!

Instead, I put it in CELERYD_OPTS and it works like a charm.:

CELERYD_OPTS="--config=backend.async.celeryconfig"

The other thing is that VIRTUAL_ENV variable is ignored as well in celeryd script. On the other celerybeat scirpt activates the virtual env, so my adviced config is:

CELERYBEAT_OPTS="--schedule=/var/run/celerybeat-schedule --config=backend.async.celeryconfig"

VIRTUAL_ENV="/path/to/.virtualenvs/your_env"

# Python interpreter from environment.
ENV_PYTHON="$VIRTUAL_ENV/bin/python"

# How to call "celeryd-multi"
CELERYD_MULTI="$VIRTUAL_ENV/bin/celeryd-multi"

#CELERYD="$VIRTUAL_ENV/celeryd"

# How to call "celeryctl"
CELERYCTL="$VIRTUAL_ENV/bin/celeryctl"

# How to call "celerybeat"
CELERYBEAT="$VIRTUAL_ENV/bin/celerybeat"