This query is related to my Google Summer of Code project. Basically, I have a master node which delegates build tasks to a number of specified image building nodes. Right now, I do it using a manual, ‘scp’, folllowed by remote execution using ‘ssh’ strategy. So, I do not have any scheduling or task distribution – automagically.
I am now looking to use a solution which might be better than my current strategy. I was looking at ‘celery’ and have played aroud with it just a bit. I have been able to run a simple job remotely. However, from what I understand so far, celery needs the source files at the worker nodes. I have also not been able to explore whether celery would do a automatic task distribution/scheduling? I am using RabbitMQ as the broker.
I am new to this, so anyone more experienced would be definitely be able to suggest whether celery is a good choice or is there anything better suitable for this?
I was a little confused. And I have it figured out. Celery will automatically choose one of the workers using the same broker. So, yes, it does the scheduling and distribution.