Home/ Questions/Q 4094898
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T19:48:41+00:00

I have a website that requires using Nokogiri on many different websites to extract

  • 0

I have a website that requires using Nokogiri on many different websites to extract data. This process is ran as a background job using the delayed_job gem. However it takes around 3-4 seconds per page to run because it has to pause and wait for other websites to respond.
I am currently just running them by basically saying

Websites.all.each do |website|
  # screen scrape
end

I would like to execute them in batches rather than one each so that I dont have to wait for a server response from every site (can take up to 20 seconds on occassion).

What would be the best ruby or rails way to do this?

Thanks for your help in advance.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You need to use delayed job. Check out this Railscasts.

    Keep in mind most hosts charge for this type of thing.

    You can also use the spawn plugin if you don’t care about managing threads but it is much much easier!!!

    This is literally all you need to do:

    1. rails plugin/install https://github.com/tra/spawn.git
    2. Then in your controller or model add the method

    For example:

     spawn do
    #execute your code here :)
 end

    http://railscasts.com/episodes/171-delayed-job

    https://github.com/tra/spawn

    • 0