Question

In Celery 2, I had a TaskSet that was set up like the following:

for (item,jobId) in itemsAndJobs:
    tasks.append(waitForOutput.subtask((jobId,item)))
job = TaskSet(tasks)
result = job.apply_async()

And then later on, I would check to see if the TaskSet had completed by checking:

job.ready() and job.successful()

This worked fine- the waitForOutput tasks would slowly move along, and then when they were all done the job check would complete. I could check multiple times without any problems.

In celery 3, I tried the quick and dirty way of changing this to a group by just changing

TaskSet(tasks)

to

group(tasks)

Unless I wait until all of the waitForOutput tasks to complete before I ever check ready and successful, this never works. ready() always returns false. I added some logging and a default retry of 30 seconds, and this is what I see-

• start 5 waitForOutput jobs
• check ready(), no waitForOutput jobs have completed, ready is false
• 2 waitForOutput jobs complete
• check ready(), ready is false, job.check_completed() is 2
• check ready(), ready is false, job.check_completed() is 0
• remaining 3 waitForOutput jobs complete
• check ready(), ready is false, job.check_completed() is 3

I see the same behavior if I take my Celery 3 code and import TaskSet from task and use that instead of group.

I would love to be told I’m just using groups incorrectly!

Answer 1

I solved this by moving to Redis as my result backend. It appears to be a bug with using AMQP as a result backend.