This problem is intermittent, occurring every few hours on a process that runs every 30 seconds and performs the following network communication:
- There is a server on a remote machine listening
- The client connects to this machine and tells the server to connect back to it
- The server closes the socket and reconnects to the client
- When the socket closes on the client side, the client immediately starts listening on the same port
The JVM_Bind error happens on the client when it tries to start listening on the same port, even though it only tries to do this after the original socket is closed. Adding a slight delay of 100 milliseconds between the socket closing and opening a new socket prevents the JVM_Bind error.
What tools can I use to debug this scenario? Running continuous netstat has not shown anything interfering with the port.
Even though it’s a kind of bad idea to start a new socket listening on the same port, why would this error occur intermittently?
EDIT : Some additional information. The server connects back to the client from an ephemeral port on the server. The Client’s old socket would go straight into CLOSED and should be able to be re-opened. Wouldn’t the likely cause be something happening on the OS, JVM or hardware?
After messing around with this issue we eventually ran Wireshark and Procmon to monitor the TCP connections and packets. Procmon is a really great tool for debugging problems on Windows and can be configured to show when TCP connections are created and dropped. Wireshark shows the actual packet data going across the line. This output showed that things were proceeding as they should have when the JVM_Bind error occurs. There was no material difference in the logs between when it worked and when it failed.
This pointed the blame at the system. I thought about system settings and then turned my attention to the possibility of a driver problem. Updated the network adapter drivers and the problem has gone away.
Moral of the story is to debug network problems with Wireshark and Procmon, it turned this problem into an analysable one.