Question

Many (>1000) workers (process) do some and want to save their work results in database. Result of work is JSON object. Workers produce JSON objects 1-5 per second. Database saver is separated process. Unidirectional connections for transfer JSON object from worker to saver is multiprocessing.Pipe. Number of pipes equal number of workers.

In saver process periodically сall:

def recv_data(self):
    data = []
    for pipe in self.data_pipe_pool:
        if pipe.poll():
            data.append(pipe.recv())
    return data

self.data_pipe_pool – list of pipes from workers.

All work fine if i run ~100 workers. If i run >1000 workers i get exception:

2013-02-13T15:17:40.731429
Traceback (most recent call last):
  File "saver.py", line 44, in run
    profile = self.poll_data()
  File "saver.py", line 116, in poll_data
    ret = self.recv_data()
  File "saver_unit.py", line 127, in recv_data
    if pipe.poll():
IOError: handle out of range in select()

I know that this is due with select() call and that:

FD_SETSIZE is usually defined to 1024 in GNU/Linux systems

But where called select? If in pipe.poll(), why I exceed the FD_SETSIZE limit, i’m calling pipe.poll() for 1 pipe individually? Where can i watch python language sources with this call select?

What workaround not exceed FD_SETSIZE limit or not use select?

Answer 1

If you check the select manual page, you will see that:

Executing FD_CLR() or FD_SET() with a value of fd that is negative or is equal to or larger than FD_SETSIZE will result in undefined behavior.

This means that if select is used behind the scene in your poll call (which seems likely), and you have file descriptors larger than FD_SETSIZE (very likely if you have over 1000 pipes) then the result can be anything.