Question

I have encountered a problem of socket communication on linux system, the communication process is like below: client send a message to ask the server to do a compute task, and wait for the result message from server after the task completes.

But the client would hangs up to wait for the result message if the task costs a long time such as about 40 minutes even though from the server side, the result message has been written to the socket to respond to the client, but it could normally receive the result message if the task costs little time, such as one minute. Additionally, this problem only happens on customer environment, the communication process behaves normally in our testing environment.

I have suspected the cause to this problem is the default timeout value of socket is different between customer environment and testing environment, but the follow values are identical on these two environment, and both Client and server.

getSoTimeout:0
getReceiveBufferSize:43690
getSendBufferSize:8192
getSoLinger:-1
getTrafficClass:0
getKeepAlive:false
getTcpNoDelay:false

the codes on CLient are like:

Message msg = null;
ObjectInputStream in = client.getClient().getInputStream();
//if no message readObject() will hang here
while ( true ) {
  try {
   Object recObject = in.readObject();
   System.out.println("Client received msg.");
   msg = (Message)recObject; 
   return msg;
       }catch (Exception e) {
    e.printStackTrace();
    return null;
   }
}

the codes on server are like,

ObjectOutputStream socketOutStream = getSocketOutputStream();
try {
  MessageJobComplete msgJobComplete = new MessageJobComplete(reportFile, outputFile );
  socketOutStream.writeObject(msgJobComplete);
  }catch(Exception e) {
    e.printStackTrace();
  }

in order to solve this problem, i have added the flush and reset method, but the problem still exists:

ObjectOutputStream socketOutStream = getSocketOutputStream();
try {
   MessageJobComplete msgJobComplete = new MessageJobComplete(reportFile, outputFile );
   socketOutStream.flush();
   logger.debug("AbstractJob#reply to the socket");
   socketOutStream.writeObject(msgJobComplete);
   socketOutStream.reset();
   socketOutStream.flush();
   logger.debug("AbstractJob#after Flush Reply");
 }catch(Exception e) {
    e.printStackTrace();
            logger.error("Exception when sending MessageJobComplete."+e.getMessage());
 }

so do anyone knows what the next steps i should do to solve this problem.
I guess the cause is the environment setting, but I do not know what the environment factors would affect the socket communication?

And the socket using the Tcp/Ip protocal to communicate, the problem is related with the long time task, so what values about tcp would affect the timeout of socket communication?

After my analysis about the logs, i found after the message are written to the socket, there were no exceptions are thrown/caught. But always after 15 minutes, there are exceptions in the objectInputStream.readObject() codes snippet of Server Side which is used to accept the request from client. However, socket.getSoTimeout value is 0, so it is very strange that the a Timed out Exception was thrown.

{2012-01-09  17:44:13,908} ERROR java.net.SocketException: Connection timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:146)
   at sun.security.ssl.InputRecord.readFully(InputRecord.java:312)
   at sun.security.ssl.InputRecord.read(InputRecord.java:350)
   at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:809)
   at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:766)
   at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
   at sun.security.ssl.AppInputStream.read(AppInputStream.java:69)
   at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
   at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
   at  java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)

so why the Connection Timed out exceptions are thrown?

Answer 1

This problem is solved. using the tcpdump to capture the messages flows. I have found that while in the application level, ObjectOutputStream.writeObject() method was invoked, in the tcp level, many times [TCP ReTransmission] were found.

So, I concluded that the connection is possibly be dead, although using the netstat -an command the tcp connection state still was ESTABLISHED.

So I wrote a testing application to periodically sent Testing messages as the heart-beating messages from the Server. Then this problem disappeared.