Question

I use Requests and some URL’s give a different response code than a call with the same URL in a browser.

For example:

import requests
req = requests.get('http://www.jordaaninfo.nl')
responsecode = req.status_code
# responsecode == 500

gives a responsecode 500 while the page http://www.jordaaninfo.nl is available via Chrome/FF/Safari.

Thanks,
Niels

Answer 1

In my Firefox the 500 status code also occurs for http://www.jordaaninfo.nl, but the page is displayed anyway. This can be seen in the Net tab of Firebug.
My Chromium browser on Linux gets a 200 OK.

The problem (and solution) is in the Accept-Language header in the request.
Apparently, the server has special needs in this respect 😉

The problem and solution can be demonstrated with wget:

$ wget 'http://www.jordaaninfo.nl/'
--2012-11-20 23:15:20--  http://www.jordaaninfo.nl/
Resolving www.jordaaninfo.nl... 87.250.155.10
Connecting to www.jordaaninfo.nl|87.250.155.10|:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2012-11-20 23:15:22 ERROR 500: Internal Server Error.

$ wget --header="Accept-Language: Accept-Language: en;q=0.8,en-us;q=0.5" 'http://www.jordaaninfo.nl/'
--2012-11-20 23:16:01--  http://www.jordaaninfo.nl/
Resolving www.jordaaninfo.nl... 87.250.155.10
Connecting to www.jordaaninfo.nl|87.250.155.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
....

The 500 Error in my Firefox was caused by language ‘nl’ as first entry in the Accept-Language setting:

nl,en;q=0.8,en-us;q=0.5,fy;q=0.3 --> 500 ERROR
en,nl;q=0.8,en-us;q=0.5,fy;q=0.3 --> 200 OK