Question

I’m using tornado to fetch many web pages asyncroniously using HTTP proxy. So, many of my fetch’es done with errors (my proxies are unreliable). I want to retry them instantly with another proxy. Here is example:

from tornado import ioloop
from tornado import httpclient

def handle_request(response):
    if response.error:
        print "Error:", response.error
        // HERE i want to put my retry with another proxy
    else:
        print response.body
    ioloop.IOLoop.instance().stop()

http_client = httpclient.AsyncHTTPClient()
http_client.fetch("http://www.google.com/", handle_request)
ioloop.IOLoop.instance().start()

But how i can i add new fetch event to current loop from handle_request ? Also, how can i pass variables to handle_request (list with all my proxies).

Answer 1

You’re asking two questions –

I would consider the use of partials http://docs.python.org/library/functools.html#partial-objects

from functools import partial

PROXIES = [A, B, C, D] # As appropriate
...
def handle_request(proxies, response):
    if ...BAD RESPONSE...:
        return http_client.fetch(response.request.url, partial(handle_request, proxies[1:]))
    # Now handle the case that you have a good result or you're out of proxies

http_client.fetch("http://www.google.com/", partial(handle_request, PROXIES[:]))

The other choice of course is to make it an object.

class ProxyRequest(object):
     PROXIES = [A, B, C]

     def __init__(self, url):
          self.url = url
          self.proxies = self.PROXIES[:]
          self.fetch()

     def fetch(self):
          p, self.proxies = self.proxies[0], self.proxies[1:]

          http_client.fetch(self.url, self.handle, proxy=p)

     def handle(self, response):
          if response.error:
               if self.proxies:
                     return self.fetch()
               else:
                     ...error case...

          ...stop the ioloop if you want...