Question

I am currently using Redis for a workflow with a couple of steps in it. For each step, a worker snatches the payload from a queue, and when its done, pushes it onto the next queue, where the next worker can take it further. If an exception occurs, the task is put into a special queue by the worker.

The application logic with regards to the flow through the application hence lies in the workers themselves. I now want to switch to Celery.

I understand that in Celery you can use subtasks, but I fail to see how you express your specific error handling there for different conditions such as exceptions and time-outs. Are you supposed to use different queues or use subtasks, and what would that look like in code?

Answer 1

I have now read the docs even more thoroughly and additionally made some tests, and this works:

The problem is to string together tasks so that they happen one after the other, but at the same time be able to handle error conditions and “break out” of the flow and do something else, not just abort.

You can string together tasks with link, and if an extra parameter *link_error* is in there it will be used for failure. From reading:

http://docs.celeryproject.org/en/latest/userguide/calling.html#linking-callbacks-errbacks
I made this:

res = add.apply_async((2, 2), link=mul.s(16), link_error=onerror.s())

The three tasks are add, mul and onerror. Add adds two numbers and mul multiplies two numbers. So this will add 2 and 2 together, and then the sum will carried over to the next step (mul) and be multiplied by 16.
However, if the add code is buggy, or has bad data or if something else bad but detectable occurs, add throws an exception and the onerror task will be run instead of mul. The onerror task gets the uuid of the job and can look the job up in the database backend, if such a one is configured. The onerror task can then archive the job or send an e-mail or whatever.