Question

This is a two part question. The first is how do you properly close a connection with urllib2? I have seen a number of examples and I have adopted the best solution I could find. However, there appears to be a problem closing the files.

Currently I use contextlib’s closing() as follows:

    try:
        with closing(self.opener.open(self.address, 
                                      None, 
                                      self.timeout)) as page:
            self.data = page.read()
    except:
        # bail out..

However, I still get “too many open files” error after a long time on OSX. I used ulimit to increase files to 2000 and above. I also set the kernel’s max files up to >40,000. I should note, the object this method is located is not disposed of and it remains around for the life of the program. However, I only keep the “data” stored in the object along with the address and timeout. I don’t keep the file-like object stored. I thought the problem may be references but I don’t believe so because I never store a reference to the file-like object directly, only the data from read(). These objects are reused and reloaded with new data each time a thread pulls a url out of the Queue.

I only open roughly 50 connections at a time. I don’t quite understand how I could run out of files. Also, when I run out of files netstat begins to crap out with malloc errors:

  netstat(439) malloc: *** mmap(size=18446744073708605440) failed (error code=12)
  *** error: can't allocate region
  *** set a breakpoint in malloc_error_break to debug
  netstat: malloc 18446744073708605396 bytes: Cannot allocate memory

I also can’t find a way to reset the connections and get netstat back to normal without shutting down.

netstat -m

$ netstat -m
475/3803 mbufs in use:
    475 mbufs allocated to data
    3328 mbufs allocated to caches
407/3814 mbuf 2KB clusters in use
0/577 mbuf 4KB clusters in use
0/12 mbuf 16KB clusters in use
11242 KB allocated to network (8.3% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to drain routines

I am having trouble locating the error but I believe the connections are not being closed in a timely manner and I am well aware the connections are not being re-used even when connecting to a single domain (I would like that). That is the second part of the question. How can someone reuse a connection with urllib2?

I have multiple threads getting urls from a Queue and each retrieve the data via this kind of routine. If possible, I would like to reuse the connection if it has been opened by another thread. The only data shared between threads is the URL queue. I have looked at other modules but they appear to need more sharing of data than just a url.

Answer 1

I would recommend dropping urllib2, and trying out the fantastic Requests library.

It automatically takes care of reusing and closing connections. (docs)

You might also be interested in its support for making asynchronous requests. (docs)