Question

I’m using Ruby 1.9.3 and trying to write a Google Play scraper loosely based on this one. I am having a really hard time with the HTTPS part of it.

Basically, using Nokogiri::HTML(open("https://play.google.com/store/#{type}/details?id=#{id}")) (as in the original gem) failed on Windows, for reasons explained on this thread.

So, I tried implementing the solution from that same thread, but it is really not working at all. I’ve even stopped trying with HTTPS for now, because there must be something basic I am missing on even just HTTP.

Here’s the code I currently have:

  url = URI.parse( "http://google.com/" )
  http = Net::HTTP.new( url.host, url.port )
  http.use_ssl = true if url.port == 443
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  res, data = http.get ("http://google.com/")
  puts data

In this case, I get nothing. Not even “nil”, just no output at all.

However, when I just do a straight Net::HTTP.get_print URI('http://www.google.com'), I get the output, no problems.

Any help would be most appreciated. The real solution I am looking for is a simple way to scrape Google Play pages when using Windows — this is just a step on the way there. So, if you know of a simpler way to accomplish this, I’d love to hear about it.

Answer 1

The reason you are getting nil is because data doesn’t have anything assigned to it. This line is only assigning to res:

res, data = http.get("http://google.com/")

Also, Google must be accessed using http://www.google.com with the www otherwise all you get back is a 301 redirect message and Net::HTTPMovedPermanently object.