I am planning on writing a rails application which will distribute tasks to several workers on different computers. I am trying to review the different technologies/gems available to use and I’m having trouble deciding which combination best suites my needs.
Here’s what I know so far; The application will need to be fault tolerant and not lose tasks (even if the worker is forcibly turned off). The number of tasks isn’t high, but the amount of time each task will take will vary (it could be relatively short or may take a long time).
I’ve seen Celery for python do something very close to what I need, but I’d like to give this a try in Ruby where it can requeue tasks in RabbitMQ if any worker is killed off. Any information would be greatly appreciated. Thank you!
This is not something I know a lot about, but have you looked at Resque?