Question

The following piece of code works as expected when running in a local install of django apache 2.2

 fx = urllib2.Request(f);
 fx.add_header('User-Agent','Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.36 Safari/525.19');
 url_opened = urllib2.urlopen(fx);

However when I enter that code into IDLE on the same machine I get the following error:

    url_opened = urllib2.urlopen(fx);
  File "C:\Python25\lib\urllib2.py", line 124, in urlopen
    return _opener.open(url, data)
  File "C:\Python25\lib\urllib2.py", line 387, in open
    response = meth(req, response)
  File "C:\Python25\lib\urllib2.py", line 498, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python25\lib\urllib2.py", line 425, in error
    return self._call_chain(*args)
  File "C:\Python25\lib\urllib2.py", line 360, in _call_chain
    result = func(*args)
  File "C:\Python25\lib\urllib2.py", line 506, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 407: Proxy Authentication Required

Any ideas?

Answer 1

urllib and urllib2 I think look at environment variables for proxies if one isn’t set programatically. Maybe the proxy environment variables haven’t been set properly in IDLE?

Compare the output of the following from IDLE to the Django program:

import os, pprint
for k in os.environ:
    if 'proxy' in k.lower(): # look for proxy environment variables
        print k, os.environ[k]

EDIT: Quoting http://docs.python.org/library/urllib2.html#urllib2.ProxyHandler:

Cause requests to go through a proxy. If proxies is given, it must be a 
dictionary mapping protocol names to URLs of proxies. The default is to read the 
list of proxies from the environment variables. If no proxy environment 
variables are set, in a Windows environment, proxy settings are obtained from 
the Internet Settings section and in a Mac OS X environment, proxy 
information is retrieved from the OS X System Configuration Framework.

To disable autodetected proxy pass an empty dictionary.

Maybe Django creates a ProxyHandler? Try calling urllib2.ProxyHandler() in IDLE.