Home/ Questions/Q 8969857
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-15T17:39:08+00:00

I’m using open-uri and nokogiri with ruby to do some simple webcrawling. There’s one

  • 0

I’m using open-uri and nokogiri with ruby to do some simple webcrawling.
There’s one problem that sometimes html is read before it is fully loaded. In such cases, I cannot fetch any content other than the loading-icon and the nav bar.
What is the best way to tell open-uri or nokogiri to wait until the page is fully loaded?

Currently my script looks like:

require 'nokogiri'
require 'open-uri'

url = "https://www.the-page-i-wanna-crawl.com"
doc = Nokogiri::HTML(open(url, ssl_verify_mode: OpenSSL::SSL::VERIFY_NONE)) 
puts doc.at_css("h2").text
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    What you describe is not possible. The result of open will only be passed to HTML after the open method as returned the full value.

    I suspect that the page itself uses AJAX to load its content, as has been suggested in the comments, in this case you may use Watir to fetch the page using a browser

    require 'nokogiri'
require 'watir'

browser = Watir::Browser.new
browser.goto 'https://www.the-page-i-wanna-crawl.com'

doc = Nokogiri::HTML.parse(browser.html)

    This might open a browser window though.

    • 0