Question

I have a Sinatra app I plan on hosting on Heroku.

This application, in part, scrapes a lot of information from other pages around the net and stores the information to a database. These scrapping operations are a slow process, so I need them to run in another thread/process separate from my Sinatra app.

My plan is just to have a button for each process that I can click and the scrapping will take place in the background.

I’m unsure what’s the best way to do this, complicated by what Heroku will allow.

Answer 1

There’s a gem called hirefire specifically for that:

HireFire automatically “hires” and “fires” (aka “scales”) Delayed Job
and Resque workers on Heroku. When there are no queue jobs, HireFire
will fire (shut down) all workers. If there are queued jobs, then
it’ll hire (spin up) workers. The amount of workers that get hired
depends on the amount of queued jobs (the ratio can be configured by
you). HireFire is great for both high, mid and low traffic
applications. It can save you a lot of money by only hiring workers
when there are pending jobs, and then firing them again once all the
jobs have been processed. It’s also capable to dramatically reducing
processing time by automatically hiring more workers when the queue
size increases.