Question

I have a celery task that makes a subprocess call that can only be run parallel system-wide. While I figure out a better solution, I need to be able to execute tasks that run serially on a single worker.

I know I could create a table to track which server is currently running the task, and wait for that to finish before future tasks are run, but it seems like there must be a better way. I’ve looked pretty extensively into the celery documentation, and briefly at the rabbitmq documentation to no avail.

Has anyone out there come up with an elegant solution to this problem that hooks into some hidden features of the aforementioned technologies?

Answer 1

If you setup a database cache in django that uses a database common to all servers you could implement this celery cookbook solution for executing one task at time. Instead of using a variable lock_id, just use a constant.

In the event that the lock is not available to the worker executing the task, call retry to put the task back into the queue.