Question

Starting with urllib2 and feedparser libraries in Python I’m getting the following error most of the time whenever try to connect and fetch content from particular URL:

urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

The minimal reproducible examples (basic, using feedparser.parser directly and advanced, where I use urllib2 library first to fetch XML content) are pasted below.

# test-1
import feedparser
f = feedparser.parse('http://www.zurnal24.si/index.php?ctl=show_rss')
title = f['channel']['title']
print title

# test-2
import urllib2
import feedparser
url = 'http://www.zurnal24.si/index.php?ctl=show_rss'
opener = urllib2.build_opener()
opener.addheaders = [('User-Agent', 'Mozilla/5.0')]
request = opener.open(url)
response = request.read()
feed = feedparser.parse(response)
title = feed['channel']['title']
print title

When I try with different URL addresses (e.g., http://www.delo.si/rss/), everything works fine. Please note that all URL’s lead to non-english (i.e., Slovenian) RSS feeds.

I run my experiments both from local and remote machine (via ssh). The error reported occurs more frequently on remote machine, although it’s unpredictable even on local host.

Any suggestions would be greatly appreciated.

Answer 1

How often does the timeout occur? If it’s not frequent, you could wait after each timeout and then retry the request:

import urllib2
import feedparser
import time
import sys

url = 'http://www.zurnal24.si/index.php?ctl=show_rss'
opener = urllib2.build_opener()
opener.addheaders = [('User-Agent', 'Mozilla/5.0')]

# Try to connect a few times, waiting longer after each consecutive failure
MAX_ATTEMPTS = 8
for attempt in range(MAX_ATTEMPTS):
    try:
        request = opener.open(url)
        break
    except urllib2.URLError, e:
        sleep_secs = attempt ** 2
        print >> sys.stderr, 'ERROR: %s.\nRetrying in %s seconds...' % (e, sleep_secs)            
        time.sleep(sleep_secs)

response = request.read()
feed = feedparser.parse(response)
title = feed['channel']['title']
print title