Question

First off, my problem is quite similar to this one. I would like a timeout of urllib.urlopen() to generate an exception that I can handle.

Doesn’t this fall under URLError?

try:
    response = urllib.request.urlopen(url, timeout=10).read().decode('utf-8')
except (HTTPError, URLError) as error:
    logging.error(
        'Data of %s not retrieved because %s\nURL: %s', name, error, url)
else:
    logging.info('Access successful.')

The error message:

resp = urllib.request.urlopen(req, timeout=10).read().decode(‘utf-8’)
File “/usr/lib/python3.2/urllib/request.py”, line 138, in urlopen
return opener.open(url, data, timeout)
File “/usr/lib/python3.2/urllib/request.py”, line 369, in open
response = self._open(req, data)
File “/usr/lib/python3.2/urllib/request.py”, line 387, in _open
‘_open’, req)
File “/usr/lib/python3.2/urllib/request.py”, line 347, in _call_chain
result = func(*args)
File “/usr/lib/python3.2/urllib/request.py”, line 1156, in http_open
return self.do_open(http.client.HTTPConnection, req)
File “/usr/lib/python3.2/urllib/request.py”, line 1141, in do_open
r = h.getresponse()
File “/usr/lib/python3.2/http/client.py”, line 1046, in getresponse
response.begin()
File “/usr/lib/python3.2/http/client.py”, line 346, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.2/http/client.py”, line 308, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.2/socket.py”, line 276, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

There was a major change from in Python 3 when they re-organised the urllib and urllib2 modules into urllib. Is it possible that there was a change then that causes this?

Answer 1

Catch the different exceptions with explicit clauses, and check the reason for the exception with URLError (thank you Régis B. and Daniel Andrzejewski)

from socket import timeout
from urllib.error import HTTPError, URLError

try:
    response = urllib.request.urlopen(url, timeout=10).read().decode('utf-8')
except HTTPError as error:
    logging.error('HTTP Error: Data of %s not retrieved because %s\nURL: %s', name, error, url)
except URLError as error:
    if isinstance(error.reason, timeout):
        logging.error('Timeout Error: Data of %s not retrieved because %s\nURL: %s', name, error, url)
    else:
        logging.error('URL Error: Data of %s not retrieved because %s\nURL: %s', name, error, url)
else:
    logging.info('Access successful.')

NB For recent comments, the original post referenced python 3.2 where you needed to catch timeout errors explicitly with socket.timeout. For example

    # Warning - python 3.2 code
    from socket import timeout
    
    try:
        response = urllib.request.urlopen(url, timeout=10).read().decode('utf-8')
    except timeout:
        logging.error('socket timed out - URL %s', url)