Question

I’m trying to read a URL within our corporate network. Spesifically the server I’m contacting is in one office and the client PC is in another:

print(urlopen(r"http://london.mycompany/mydir/").read())

Whenever I run this function I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
File "C:\Python24\lib\urllib2.py", line 130, in urlopen
  return _opener.open(url, data)
File "C:\Python24\lib\urllib2.py", line 364, in open
  response = meth(req, response)
File "C:\Python24\lib\urllib2.py", line 471, in http_response
  response = self.parent.error(
File "C:\Python24\lib\urllib2.py", line 402, in error
 return self._call_chain(*args)
File "C:\Python24\lib\urllib2.py", line 337, in _call_chain
 result = func(*args)
File "C:\Python24\lib\urllib2.py", line 480, in http_error_default
 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required

The odd thing is that there’s no firewall between these two computers – for some reason url has decided to connect to the web-server via the proxy which we’d normally use to connect to content outside the company, and in this case that’s failing because I’ve not authenticated it.

I’m pretty sure that the fault occurs within the client PC: I did a nslookup and a ping to the server to confirm that there’s a connection between the two computers, however when I watch the transaction using TCPView for Windows I can see that the python.exe process is connecting to a completely different server (yes, the proxy!).

So what could be causing this? Note that the os.environ[“http_proxy”] variable is NOT set – this variable is often used to make urllib connect via a proxy server. That’s not the case here. Could there be something else which might have the same effect?

FYI, Running Python 2.4.4 on Windows XP 32bit in a very locked-down corporate environment.

Answer 1

It reads from the system settings. Use urllib.FancyURLOpener:

opener = urllib.FancyURLopener({})
f = opener.open("http://london.mycompany/mydir/")
f.read()