Question

I try to do simple async http client with asyncore:
This code works fine and output is (fast enought):

www.gmail.com  :  recv http code:  301
www.yandex.ru  :  recv http code:  200
www.python.org  :  recv http code:  200
www.google.ru  :  recv http code:  200
www.gravatar.com  :  recv http code:  302
www.com.com  :  recv http code:  302
www.yahoo.com  :  recv http code:  302
www.bom.com  :  recv http code:  301

But than i uncomment line with not exist host:

#c = AsyncHTTP('http://www.no-such-host.ru') #!this line breaks execution!

The execution breaks, code hangs for some time, output part of data and hangs with no last data output:

connection error: [Errno -5] No address associated with hostname
www.gmail.com  :  recv http code:  301
www.yandex.ru  :  recv http code:  200
www.yahoo.com  :  recv http code:  302
www.com.com  :  recv http code:  302
www.bom.com  :  recv http code:  301
www.gravatar.com  :  recv http code:  302

…
some hosts are lost here and long delay at start.

Why this happen and how to fix this?

# coding=utf-8
import asyncore
import string, socket
import StringIO
import mimetools, urlparse

class AsyncHTTP(asyncore.dispatcher):
    # HTTP requestor

    def __init__(self, uri):
        asyncore.dispatcher.__init__(self)

        self.uri = uri


        # turn the uri into a valid request
        scheme, host, path, params, query, fragment = urlparse.urlparse(uri)
        assert scheme == "http", "only supports HTTP requests"
        try:
            host, port = string.split(host, ":", 1)
            port = int(port)
        except (TypeError, ValueError):
            port = 80 # default port
        if not path:
            path = "/"
        if params:
            path = path + ";" + params
        if query:
            path = path + "?" + query

        self.request = "GET %s HTTP/1.0\r\nHost: %s\r\n\r\n" % (path, host)

        self.host = host
        self.port = port

        self.status = None
        self.header = None
        self.http_code = None
        self.data = ""

        # get things going!
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        #self.connect((host, port))
        #return
        try:
            self.connect((host, port))
        except Exception,e:
            self.close()
            self.handle_connect_expt(e)

    def handle_connect(self):
        self.send(self.request)

    def handle_expt(self):
        print "handle_expt error!"
        self.close()

    def handle_error(self):
        print "handle_error error!"
        self.close()

    def handle_connect_expt(self,expt):
        print "connection error:",expt

    def handle_code(self):        
        print self.host," : ","recv http code: ",self.http_code


    def handle_read(self):
        data = self.recv(2048)
        #print data
        if not self.header:
            self.data = self.data + data
            try:
                i = string.index(self.data, "\r\n\r\n")
            except ValueError:
                return # continue
            else:
                # parse header
                fp = StringIO.StringIO(self.data[:i+4])
                # status line is "HTTP/version status message"
                status = fp.readline()
                self.status = string.split(status, " ", 2)
                self.http_code = self.status[1]
                self.handle_code()      

                # followed by a rfc822-style message header
                self.header = mimetools.Message(fp)
                # followed by a newline, and the payload (if any)
                data = self.data[i+4:]
                self.data = ""
                #header recived
                #self.close()


    def handle_close(self):
        self.close()




c = AsyncHTTP('http://www.python.org')
c = AsyncHTTP('http://www.yandex.ru')
c = AsyncHTTP('http://www.google.ru')
c = AsyncHTTP('http://www.gmail.com')
c = AsyncHTTP('http://www.gravatar.com')
c = AsyncHTTP('http://www.yahoo.com')
c = AsyncHTTP('http://www.com.com')
c = AsyncHTTP('http://www.bom.com')
#c = AsyncHTTP('http://www.no-such-host.ru') #!this line breaks execution!
asyncore.loop()

ps: My system ubuntu 11.10 + python 2.7.2

Answer 1

You invoke a blocking name-resolution when you do self.connect((host, port)). Combined with your local DNS configuration, this is why your program has a long delay at startup.

An alternative to asyncore and figuring out how to do non-blocking name resolution yourself, you might think about using Twisted. Twisted’s TCP connection setup API (mainly reactor.connectTCP or one of the APIs built on top of it) does not block. So a naive use of it will remain properly asynchronous.