Question

I am trying to scrape some information of a page using the jsdom.env function. However, the page that gets returned in the env() callback is about how access is denied to the server instead of the content that I am hoping to see when I load the same URL in a browser.

Thus, there seems to be a difference in how the browser loads the page vs. how jsdom is loading it. Is this something which can be configured in the jsdom module?

Edit:

Example URL: http://www.bestbuy.com/site/HP+-+20%22+Widescreen+Flat-Panel+LCD+Monitor/1422209.p?id=1218257754431&skuId=1422209

Update:

The issue was jsdom not specifying the user-agent http header. Look at the detailed answer below

Answer 1

The problem was that jsdom is not specifying a ‘User-Agent’ http header, which the bestbuy.com server are checking for. If its empty, access is denied. Currently, there is no way of specifying this through jsdom – https://github.com/tmpvar/jsdom/issues/196

A workaround that worked for me to use the request module to get the page content and then pass then on to jsdom to work on. The request module allows you to specify a user agent

Example:

var request = require('request'),

getPage = function(someUri, callback) {
  request({uri: someUri, headers:{'User-Agent': 'Mozilla/5.0'}}, function (error, response, body) {
    console.log("Fetched " +someUri+ " OK!");
    callback(body);
  });
}

getPage('http://www.bestbuy.com/', function(body) {
   console.log(body)
});