Question

How can I gracefully shutdown a Java SSL session without closing the underlying socket?

The scenario is that a Java client connects to a (non-java) server, sets up SSL and securely sends credentials (user name & password) to the server. The server sets up an environment running under these credentials, spawns a new process in this environment and passes the socket handle to it, but the catch is that this new process can’t reuse the existing SSL connection (and can’t use SSL session resuming)… so the idea is to shutdown SSL before this new process is spawned, and then re-negotiate an SSL session from scratch with the new process.

The problem is that Java’s SSLSocket’s close() method closes the socket in addition to shutting down the SSL session (by sending the close_notify alert). There doesn’t seem to be an equivalent of OpenSSL’s SSL_shutdown() function, which allows leaving the underlying socket open.

I have tried a few things to get around this:

1. Using SSLSocket.startHandshake() a second time, but that automatically tries to resume the existing cached SSL session (which fails, since the spawned server process does not know of this session), and, while there is a method to force resuming SSL sessions or die trying, there is no method to invalidate all cached sessions or disable use of cached sessions.

2. Creating an SSLSocket over the top of my existing socket using SSLSocketFactory.createSocket(), with autoClose set to false. This does not stop the close() method from closing the underlying socket, and I suspect that the autoClose parameter only prevents the socket from being closed when the initial handshake fails.

3. Creating an SSLSocket over an existing socket (as above), then creating a second SSLSocket to use for the SSL handshake with the spawned process (from a new SSLContext). This fails because when the server sends the close_notify alert (before spawning the sub-process), Java closes the socket.

I have heard of SSLEngine, but I have also read (though the source currently eludes me) that it is a lot of painful effort to write a correct implementation of SSL over TCP/IP using it, and it seems like overkill when all I need is to have a version of SSLSocket whose close() doesn’t call super.Close().

However, SSLSocket does not appear to even override close() (according to the javadoc), so I am unsure as to how it hooks the close() method when there does not appear to be any support for registering close listeners on Socket.

Company policy dictates that no third-party encryption libraries can be used, so I cannot turn to an alternative SSL implementation such as that by Bouncy Castle to solve the problem.

If we can’t get the current 1-socket design to work, the alternative is to rewrite the client & server to use 2 separate sockets (which gets quite messy in terms of having to counter denial of service and man-in-the-middle attacks, and isn’t that what SSL is supposed to be for in the first place?).

Any input or thoughts on ways to solve this problem would be most welcome.

Answer 1

This does not stop the close() method
from closing the underlying socket

Yes it does. As per the documentation.

and I suspect that the autoClose
parameter only prevents the socket
from being closed when the initial
handshake fails.

No. It prevents closing of the underlying socket. As per the documentation.

Show us some code.