Question

I am using multiprocessing.Pool.map, which forks the current process.

My understanding is that by default, all file descriptors including sockets are copied from the master process when forking. The master process itself is a web server (using cherrypy), so this wreaks havoc with open ports etc. The forked processes are really only doing some CPU-heavy numerical stuff inside one of the libraries that the server is using — nothing to do with the web/socket part.

Is there an easy way to automatically close all sockets in the new processes? Or another way to avoid issues with forking a CherryPy server?

Using CherryPy 3.2.2, Python 2.7; must work on Linux and OS X.

Answer 1

POSIX does not include a sensible way to list or close a range of file descriptors.

So we have to loop over the full range (like from 3 to 1023), closing the file descriptors one at a time.

Or, if we have a /proc file system, we can read the list of open file descriptors in /proc/self/fd and close just those. This can be quicker than closing all possible file descriptors.

import os

def close_files(fd_min=3, fd_max=-1):
    if os.path.exists('/proc/self/fd'):
        close_files_with_procfs(fd_min, fd_max)
    else:
        close_files_exhaustively(fd_min, fd_max)

def close_files_exhaustively(fd_min=3, fd_max=-1):
    import resource
    fd_top = resource.getrlimit(resource.RLIMIT_NOFILE)[1] - 1
    if fd_max == -1 or fd_max > fd_top:
        fd_max = fd_top
    for fd in range(fd_min, fd_max+1):
        try:
            os.close(fd)
        except OSError:
            pass

def close_files_with_procfs(fd_min=3, fd_max=-1):
    for nm in os.listdir("/proc/self/fd"):
        if nm.startswith('.'):
            continue
        fd = int(nm)
        if fd >= fd_min and (fd_max == -1 or fd < fd_max):
            try:
                os.close(fd)
            except OSError:
                pass

def timereps(reps, func):
    from time import time
    start = time()
    for i in range(0, reps):
        func()
    end = time()
    return (end - start) / reps

print "close_files: %f" % timereps(100, lambda: close_files())
print "close_files_exhaustively: %f" % timereps(100, lambda: close_files_exhaustively())
print "close_files_with_procfs: %f" % timereps(1000, lambda: close_files_with_procfs())

On my system:

$ python ./close_fds.py 
close_files: 0.000094
close_files_exhaustively: 0.010151
close_files_with_procfs: 0.000039