Question

I have an asynchronous client and server application that wrap OpenSSL. Once created, they call SSL_set_connect_state()/SSL_set_accept_state() respectively to initialize their internal state. Up to this point everything is running as it should be. I’m assuming once those aforementioned states are set, hand shaking can begin. I do the following to initialize said hand shaking (snippet):

if (BIO_ctrl_pending(m_pOutBio))
{
    size_t iNumBytes = BIO_ctrl_pending(m_pOutBio);
    if (iNumBytes > 0)
    {
        CAutoPtr<byte> tTempBuf(new byte[iNumBytes]);

        // Transfer bytes from the Write BIO into the temporary buffer
        int iRet = BIO_read(m_pOutBio, tTempBuf, iNumBytes);
    }
}

Here is where it gets interesting. The BIO_ctrl_pending() call fails and when I do a SSL_get_error() on it, it returns with SSL_ERROR_WANT_READ. It then obviously doesn’t not execute my reading code. I added the following line above my code snippet:

SSL_peek(m_pSSLCon, NULL, 0);

Now when I run the code, BIO_ctrl_pending() returns the proper amount (210) and my code can then proceed to read from the BIO and initiate hand shaking.

My question is, is this some sort of bug in OpenSSL where the state doesn’t get set properly?? Or am I missing something??

Answer 1

Old question but keeps popping up when googling openssl stuff, so for reference:

Just initiate handshake instead of relying on peek/read doing it internally:
If it’s not a socket bio the handshake will fail with ssl_error_want_read and then you use ctrl_pending to get the size to read.

Something like this:

    ret = SSL_do_handshake(client_side);
      if (ret < 0)
        if (SSL_ERROR_WANT_READ !=  SSL_get_error(client_side,ret))
            cout << "ERROR";


    ret = BIO_ctrl_pending(network_side);