Question

I need to work with a TCP socket over TLS for an app I’m working on. I’ve been through dozens of examples and while I have no problem getting through the handshake, I can’t seem to read the input stream through any means (tried a lot, including readline(), reading to character array, etc). every time I try, the app freezes on that spot. If I debug, it never goes to the next line of code.

In an attempted solution, I decided to move over to using an SSLEngine, since that’s supposed to be the Java 1.5 answer to java.nio for SSL. However, I have found one example (here: http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/samples/sslengine/SSLEngineSimpleDemo.java) which is more than a little confusing to me, and I’ve not been successful implementing it. When I try, the unwrap() call yields an empty buffer, where I know (from using OpenSSL on the command line) that the service in question pushes data back down the pipe.

Suggestions are welcome, I’ve burned way too much time on this already. Here’s the relevant code:

SSLEngine engine = sslContext.createSSLEngine(uri.getHost(), uri.getPort());
            engine.setUseClientMode(true);
            engine.beginHandshake();
            SSLSession session = engine.getSession();
            int bufferMax = session.getPacketBufferSize();
            int appBufferMax = session.getApplicationBufferSize() + 50;
            ByteBuffer cTo = ByteBuffer.allocateDirect(bufferMax);
            ByteBuffer sTo = ByteBuffer.allocateDirect(bufferMax);

            ByteBuffer out = ByteBuffer.wrap(sessionId.getBytes());
            ByteBuffer in = ByteBuffer.allocate(appBufferMax);


            debug("sending secret");
            SSLEngineResult rslt = engine.wrap(out, cTo);
            debug("first result: " + rslt.toString());
            sTo.flip();
            rslt = engine.unwrap(sTo, in);
            debug("next result" + rslt.toString());

Answer 1

This implementation is missing some key pieces. Namely the handshake can bounce between several states NEED_WRAP, NEED_UNWRAP, NEED_TASK to negotiate a connection. This means you cannot just call one and then the other. You will need to loop over the states until a handshake has completed.

   while (handshaking) {
      switch (state) {
          case NEED_WRAP:
              doWrap();
              break;
          case NEED_UNWRAP:
              doUnwrap();
              break;
          case NEED_TASK:
              doTask();
              break;
        }
    }

A full working example of Java SSL and NIO

Now that said, you should be aware the SSLEngine on Android is broken. Google recommends using threads and blocking sockets according to that thread.