Using epoll_wait to manage multiple connections using multiple threads, there is a risk trying to release custom data associated with a closed socket.
Consider the following scenario, where T is the custom data :
- Data is received,
- Because of 1, thread A deblocks from epoll_wait and processes the event (access T)
- At same time, another thread B, wants to close the connection
Thread B can’t assume that T can be safely deleted, eventhough the call to close will immediatly remove the socket from the epoll.
I had the following standard idea :
Maintain a variable within T that gets incremented each time a call to write/read returns EAGAIN, and gets decremented each time the socket is ready.
When close is called, wait for that variable to go down to zero before deleting T.
The issue I experienced is that if close is called, epoll_wait does not return an indication of a cancellation of previous calls to arm the socket.
Anybody had this same problem ? How did you managed to overcome it ?
After many research, I found this recent and remarkable article :
http://lwn.net/Articles/520012/
Basically it acknowledge the issue I am describing and speaks about a possible future patch to Linux kernel that allows to extend the epoll API in a way that solves the issue.
The extension bring a new command called : EPOLL_CTL_DISABLE.
When it is issued, and by means of return value, the calling thread will know if some other thread has just been deblocked from epoll_wait upon same socket.
This can help know the safe moment of closure and release of custom data.