Question

I am trying to maintain a pool of connections to a proxy. My code looks like this:

>>> from urllib3 import PoolManager
>>> pool = PoolManager(10)
>>> pool.urlopen('GET', 'http://http-server/index.html',fields=None,headers=None,encode_multipart=False,multipart_boundary=None,proxies={'http': 'http://proxy'})

When I run this, it is failing:

> 
> Traceback (most recent call last):
>   File "<stdin>", line 1, in   <module>
>   File "urllib3/poolmanager.py", line 117, in urlopen
>     response = conn.urlopen(method, u.request_uri, **kw)
> File "urllib3/connectionpool.py", line 427, in urlopen
>     **response_kw)   
> File "urllib3/response.py", line 195, in from_httplib
>     **response_kw) 
> TypeError: __init__() got an unexpected keyword argument 'proxies'

Any idea what I am doing wrong? According to urllib3 docs, keyword args are sent to urlopen, but looks like it is not happening in this case.

Here is the link to urllib describing the usage of proxy keyword arg.

Answer 1

The docs you linked to are to Python’s urllib, which is a different library than urllib3. When you call urllib3’s urlopen, that’s not the same urlopen as urllib’s. Sorry for the confusion. 🙂

At the moment, using a proxy with urllib3 is not well-documented. If you want to explore the urllib3 code, take a look at urllib3.poolmanager.ProxyManager.

Otherwise, I suggest trying Requests which streamlines the proxies feature on top of urllib3. See: http://docs.python-requests.org/en/latest/user/advanced/?highlight=proxy#proxies

Your code would look something like this.

>>> import requests
>>> requests.get('http://http-server/index.html', proxies={'http': 'http://proxy'})

If you’d still prefer to use urllib3, then using the ProxyManager would look something like this:

>>> import urllib3
>>> http = urllib3.proxy_from_url('http://myproxy.com/')  # This returns a ProxyManager object which has the same API as other ConnectionPool objects.
>>> r = http.request('GET', 'http://http-server/index.html')

You could also create your own ProxyManager object directly, but I prefer to use the proxy_from_url shortcut.