Question

I have a java client that invokes a thread to hit a servlet and retrieves last few lines from logs at the server, and show the retrieved log lines on the client. Every once in a while, the log thread times out.
Application server is Tomcat, but the error is intermittently reproducible across both Tomcat and Websphere, with client on Windows and server on Windows. With client on Windows and server on AIX, this problem has not occurred till now. I must mention that the code was stable for quite a few iterations, and suddenly started giving these problems.

What I have tried so far

1. The log reading client invokes the thread every 0.1 sec (used a sleep). I tried increasing the sleep time to 5 sec in the code, but it did not help.

2. When creating URLConnection object, I set properties like connectTimeout and readTimeout. I don’t think readTimeout can be a cause, because that would have thrown a Socket exception.

3 I tried working with Tomcat configuration.

Connector port="9962" protocol="HTTP/1.1" connectionTimeout="200000" redirectPort="8445" acceptCount="30"

4 . The url connection is “disconnected” after use.

5 The stack trace seems to imply that the request never reached the application server, could this be because of some OS layer limits on connection. But in that case, there would have been an entry in the Event viewer of Windows.

 java.net.ConnectException: Connection timed out: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)

How would you go about diagnosing this problem? The server logs don’t show anything suspicious. The client and server do not have any other networking devices to the best of my knowledge so no proxy is required, and the firewall is switched off.
I have not used keep alive thus far.

Answer 1

It is difficult to predict what is causing this. However, your next step should be to try running a packet sniffer on the client and / or server to see if the TCP connection requests are making it to the windows machine.

If the problem occurs both with Tomcat and Websphere, that would imply that the cause is at a lower level; i.e. in the OSes TCP/IP stacks, firewall … or in the network. (And if the server is running in a virtual, it could be a drop-out in the virtual networking.)