Question

We have a requirement to build a small Sinatra app which will capture events from an external API and add them to a queue for processing by a Rails application. We could be receiving hundreds of thousands of events per day.

Given that resque rules itself out by not being able to guarantee that jobs won’t get lost, what other options are out there. We’ve looked at delayed_job and that doesn’t play well with Sinatra, so what other alternatives are there for something fast, reliable and scalable.

Answer 1

Have you looked at Beanstalk?

• http://kr.github.com/beanstalkd/
• http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/

There’s an example Sinatra/Beanstalk app on GitHub:

• https://github.com/adamwiggins/clockwork-sinatra-beanstalk

Alternatively you might want to check out RabbitMQ with ruby-amqp, but I think I’d first try the Beanstalk approach (it handles the workload you describe in your post for us):

https://github.com/ruby-amqp/amqp