Question

I’m using urllib2 open to download some web pages. Unfortunately one page is an infinite stream (a set of live video frames) and urllib2 will not timeout because the open call succeeds, while the ‘read’ call hangs forever. Example code:

res = opener.open(encoded, timeout=timeout)
log('opened', url)
contents = res.read()
log('never get here')

Any tips on avoiding/killing these connections?

Answer 1

Using the generator approach mentioned by Jacob I integrated a “kill switch”.

startime = datetime.now()
res = opener.open(url, timeout=timeout)
contents = ''
for item in res:
  contents += item
  if (datetime.now() - starttime).seconds > timeout:
    raise IOError('timeout')