Question

I’m trying to obtain the data contained within the graph on this website. I used Firebug to find the direct link to the json and used this script (which works on other direct json links). What should be returned is a set of timestamps and prices.

    import urllib2
    import simplejson
    req = urllib2.Request("http://www.grandexchangecentral.com/include/gecgraphjson.php?jsid=10350")
    opener = urllib2.build_opener()
    f = opener.open(req)
    h = simplejson.load(f)

Running this script returns urllib2.HTTPError: HTTP Error 403: Forbidden. I can get the data manually with Firebug, but I’d like to sort through multiple data sets with a script. Is there a way to get this data without receiving an HTTP error?

Answer 1

This website looks for the Referer header and tries to make sure that the request originated from one of its webpages. This isn’t a great security measure, so it’s really easy to bypass.

I’m a bit lazy right now and don’t want to look up the documentation for urllib2, so here’s a solution with the requests module:

import requests

url = 'http://www.grandexchangecentral.com/include/gecgraphjson.php?jsid=10350'
r = requests.get(url, headers={'Referer': 'www.grandexchangecentral.com'})

data = r.json