Question

I need a framework which will allow me to do the following:

• Allow to dynamically define tasks (I’ll read an external configuration file and create the tasks/jobs; task=spawn an external command for instance)

• Provide a way of specifying dependencies on existing tasks (e.g. task A will be run after task B is finished)

• Be able to run tasks in parallel in multiple processes if the execution order allows it (i.e. no task interdependencies)

• Allow a task to depend on some external event (don’t know exactly how to describe this, but some tasks finish and they will produce results after a while, like a background running job; I need to specify some of the tasks to depend on this background-job-completed event)

• Undo/Rollback support: if one tasks fail, try to undo everything that has been executed before (I don’t expect this to be implemented in any framework, but I guess it’s worth to ask..)

So, obviously, this looks more or less like a build system, but I don’t seem to be able to find something that will allow me to dynamically create tasks, most things I’ve seem already have them defined in the “Makefile”.

Any ideas?

Answer 1

I’ve been doing a little more research and I’ve stumbled upon doit which provides the core functionality I need, without being overkill (not saying that Celery wouldn’t have solved the job, but this does it better for my use case).