Environment:
RHEL 5 servers, MySQL 5.1.43, PHP 5.1.6 (using MySQLi).
Currently only available within our internal VPN network.
Servers
ServerA: Webserver
ServerB/C/D: Database server (1 master 2 slaves)
The error (on ServerA)
[Tue May 25 11:12:17 2010] [error] [client CLIENTIP] PHP Warning: mysqli::real_connect() [function.mysqli-real-connect]: (HY000/2003): Can’t connect to MySQL server on ‘ServerB’ (4) in /home/****/Database.php on line 67, referer: [website]
Problem description
It appears that at completely random times, our website is unable to connect to one of the MySQL servers – usually the Master. Except for the forementioned error message, there is nothing to be found in any of the logs as far as I can see, and most of the times the connection is succesful and everything works as it should. It’s just at completely random times, this error pops up.
There’s no firewall blocking any internal traffic, timeout value is 3 but it doesn’t take 3 seconds before it fails to connect. With the default mysql client I can connect from ServerA to ServerB,C and D and haven’t encountered a problem yet.
It appears that there’s a bug in libmysqlclient that’s only noticeable if you manually set a time-out value. Setting it to 0 seems to solve it, for now. More information about this can be found in this post.