Home/ Questions/Q 6561309
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-25T13:34:15+00:00

I am trying to use Hpricot to parse amazon mobile.website but I found that

  • 0

I am trying to use Hpricot to parse amazon mobile.website but I found that the source code I get from browser(IE, FF, Chrome and opera) is different from that parsed by Hpricot

for example:
http://www.amazon.com/gp/aw/d/0534243126

I am trying to extract the after-discount price. By looking at the source code from any browser, this is a very very easy job: doc.at(“span[@class=’dpOurPrice’]”).inner_text

However, it turns out open-uri/hpricot gets a completely different source code and the price has NO html tag on it. Could anyone tell me what is going on here?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The Amazon mobile website server propabily uses the user agent string to determine what tipe of content to return and the user agent for ruby is not what the server expected. Set it with:

    open("http://xxx.com", 'User-Agent'=>'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2') { |f| Hpricot(f) }

    to by the same as the browser so that server returns the same source it would to a browser.

    • 0