I’m running nginx with php-fpm on a high traffic site. I let nginx communicate with php-fpm over tcp/ip, both nginx and the php-fpm pools running on the same server.
When I use tcp/ip to let nginx and php-fpm pools communicate with eachother, the loading of pages takes a few (5-10) seconds before it does anything at all, and when it finally gets going, it takes no time at all for the loading to finish. Since the statuspage of php-fpm shows that the listen backlog is full, I assume it takes some time before the request is handled.
Netstat shows a lot (20k+) connections in the TIME_WAIT status, don’t know if this is related but it seemed relevant.
When I try to let nginx and php-fpm communicate over a UNIX socket, the time before the page actually loads is reduced to almost nothing, and the time before the finished page is in my browser is 1000x less. Only problem with the UNIX sockets is that it gives me a LOT of errors in the logs:
*3377 connect() to unix:/dev/shm/.php-fpm1.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 122.173.178.150, server: nottherealserver.fake, request: "GET somerandomphpfile HTTP/1.1", upstream: "fastcgi://unix:/dev/shm/.php-fpm1.sock:", host: "nottherealserver.fake", referrer: "nottherealserver.fake"
My two questions are:
does anybody know why the tcp/ip method has such a large wait before it actually seems to connect to the php-fpm backend?
why do the UNIX sockets cause problems when using this instead of tcp/ip?
What I tried:
set net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse to 1 when trying to decrease the number of TIME_WAIT connections (went down from 30k+ to 20k+)
increased the net.core.somaxconn value from the default 128 to 1024 (tried higher too but still the same error when using the UNIX sockets)
increased the max number of open files
What is probably also quite relevant: tried using lighttpd + fastcgi, has the same problem with the long time before a connection finally gets handled. MySQL is not too busy, shouldnt be the cause of the long waiting times. Disk wait time is 0% (SSD disk), so a busy disk doesn’t seem to be the culprit either.
Hope that somebody found a fix for this problem, and is willing to share 🙂
Answering my own question since the problem is solved (not sure if this is the correct way to do it).
My problem was that APC caching didnt work at all. It was installed, configured and enabled, but did not add anything to its cache. After switching from APC to Xcache, there was a huge drop in load and loadtimes. Still don’t know why APC did nothing, but at the moment im just happy that the problem is solved 🙂
Thanks for all the input from you guys!