Question

I am using tornado.httpclient.AsyncHTTPClient.fetch to fetch domains from list. When I put domains to fetch with some big interval(500 for example) all works good, but when I decrease the inerval to 100, next exception occurs time to time:

Traceback (most recent call last):
  File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/simple_httpclient.py", line 289, in cleanup
    yield
  File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/stack_context.py", line 183, in wrapped
    callback(*args, **kwargs)
  File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/simple_httpclient.py", line 384, in _on_chunk_length
    self._on_chunk_data)
  File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/iostream.py", line 180, in read_bytes
    self._check_closed()
  File "/home/crchemist/python-2.7.2/lib/python2.7/site-packages/tornado/iostream.py", line 504, in _check_closed
    raise IOError("Stream is closed")
IOError: Stream is closed

What can be the reason of this behavior? Code looks like this:

def fetch_domain(domain):
    http_client = AsyncHTTPClient()
    request = HTTPRequest('http://' + domain,
       user_agent=CRAWLER_USER_AGENT)
    http_client.fetch(request, handle_domain)


class DomainFetcher(object):
    def __init__(self, domains_iterator):
        self.domains = domains_iterator

    def __call__(self):
        try:
            domain = next(self.domains)
        except StopIteration:
            domain_generator.stop()
            ioloop.IOLoop.instance().stop()
        else:
            fetch_domain(domain)

domain_generator = ioloop.PeriodicCallback(DomainFetcher(domains), 500)
domain_generator.start()

Answer 1

note that tornado.ioloop.PeriodicCallback takes a cycle time in integer ms while the HTTPRequest object is configured with a connect_timeout and/or a request_timeout of float seconds (see doc).

“Users browsing the Internet feel that responses are “instant” when delays are less than 100 ms from click to response” (from wikipedia) See this ServerFault question for normal latency values.

IOError: Stream is closed is validly being raised to inform you that your connection timed out without completing, or more accurately, you called the callback manually on a pipe that wasn’t open yet. This is good, since it is not abnormal for latency to be > 100ms; if you expect your fetches to complete reliably you should raise this value.

Once you’ve got your timeout set to something sane, consider wrapping your fetches in a try/except retry loop as this is a normal exception that you can expect to occur in production. Just be careful to set a retry limit!

---

Since you’re using an async framework, why not let it handle the async callback itself instead of running said callback on a fixed interval? Epoll/kqueue are efficient and supported by this framework.

import ioloop

def handle_request(response):
    if response.error:
        print "Error:", response.error
    else:
        print response.body
    ioloop.IOLoop.instance().stop()

http_client = httpclient.AsyncHTTPClient()
http_client.fetch("http://www.google.com/", handle_request)
ioloop.IOLoop.instance().start()

^ Copied verbatim from the doc.

If you go this route, the only gotcha is to code your request queue so that you have a maximum open connections enforced. Otherwise you’re likely to end up with a race condition when doing serious scraping.

It’s been ~1yr since I touched Tornado myself, so please let me know if there are inaccuracies in this response and I will revise.