Question

I’m trying to get a list of stations from this webpage – https://web.barclayscyclehire.tfl.gov.uk/maps

I see that they have the list of stations in JSON structure in the javascript. So I am trying to connect to the page and then pass the data to Nokogiri to get the javascript with JSONs and then parse the JSONs individually.

To connect to HTTPS and pass data to Nokogiri I used this code available here – https://gist.github.com/1037492

require 'net/https'
require 'nokogiri'

url = "https://example.com"
url = URI.parse( url )
http = Net::HTTP.new( url.host, url.port )
http.use_ssl = true if url.port == 443
http.verify_mode = OpenSSL::SSL::VERIFY_NONE if url.port == 443
path = url.path
path += "?" + url.query unless url.query.nil?
res, data = http.get( path )

case res
  when Net::HTTPSuccess, Net::HTTPRedirection
    # parse link
    doc = Nokogiri::HTML(data)
     # do what you want ...
  else
    return "failed" + res.to_s
end

However when I try to debug in Aptana Studio 3, before the debugger can stop on any breakpoint I have, it exits giving me an invalid return error. Is there something wrong with that code ?
And is that the best way to connect to HTTPS and pass data to Nokogiri ?

Answer 1

Try like this:

require 'nokogiri'
require 'open-uri'
require 'openssl'

OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
doc = Nokogiri::HTML open(https_url)