I am currently using Resque for dealing with background jobs in my application. Now i have 5 different queues at present (it ll grow very fast). Each of them is doing works like updating Solr indexes, Real-time Notification ,scheduled newsletters, delayed emails & SMS. And currently am using Resque as a rails gem and running the Resque from rails environment.
Now i am planning to move Solr index update task and scheduled newsletters to different server since these two performing heavy operations. One approach is just copy the rails directory to the new server and run the Resque jobs from the rails environment. But i am not comfortable doing this.
Another one is creating a seperate rake app for resque tasks. But the problem is, both these tasks are heavily tied to rails models and rails templating. I am totally unsure how to proceed next.
Has anyone faced the similar problem, and how you architected the application?
We use rubber to provision our servers and deploy. It’s a plugin for Capistrano that does role-based deployment. One of the roles is “resque_worker” and any machine with that role will start up resque-pool to start processing work.
But you can do this much more simply. Just deploy your application to two different machines. Resque is designed to allow workers on different machines. As long as your second machine can access your redis server, everything will work just fine.