I am trying to create a server which I expect to have high performance demands. This question deals with the server core. What programming ideas best support fast performance?
- Do you split sockets into different threads and call blocking recv() on each?
- Do you have one thread which sits in a select() loop and then notifies another thread to process the individual ports?
- Do you have one thread which processes the select() and the response?
- Do you do 2 or 3 but with clusters of ports instead of all of them?
- Does using blocking vs nonblocking ports matter if you use select as specified above?
- What setsockopt’s improve performance: TCP_NODELAY, others?
I realize that some of these depend on the use case. For example, 6 with TCP_NODELAY off would have a negative impact if there are a lot of small packets. 3 sounds like it might be faster if the response is trivial. Any other questions that I havent thought of that affect performance would be appreciated as well.
I would start with a single-threaded approach: Use non-blocking I/O, and a fast polling mechanism like edge-triggered epoll on Linux. (Other platforms have similar technologies.) Centering everything around your polling loop simplifies the program design massively, so I would definitely throw signalfds, timerfds and eventfds in there, too. Then everything is handled by one central loop.
If and when you need to go multi-threaded, this may be as simple as running the main loop several times concurrently. If you set events to “one-shot”, they’ll be disabled from the poll until rearmed, and so the thread that processes the event can safely assume to be the only thread doing so (and re-arm the event at the end). You only need to synchronise the communication between different parts of your program, or shared data access, but a lot of synchronisation is already taken care by the poller.