Question

I am trying to use OpenSSL in my application, and achieve a secure connection.

At first I tried:

1. create ssl struct
2. create socketbio for the tcp socket
3. create a sslbio
4. set socketbio to SSL strcut
5. SSL_accept(ssl)
6. BIO_push(ssl, socketbio)

This cause handshake to happen successfully, but application data wasn’t properly decrypted.

Then I tweaked a little, and relaced 6 with

(new) BIO_ctrl(sslbio, SET_SSL, ssl)

and things worked fine.

I Wanted to know, what’s wrong with previous approach, and what’s causing the new apprach work?

Answer 1

It’s hard to answer the question without knowing why you think BIO_push is all you need to do. At any rate, you shouldn’t call BIO_ctrl directly. You should use the high-level wrapper BIO_set_ssl defined in bio.h:

#define BIO_set_ssl(b,ssl,c)    BIO_ctrl(b,BIO_C_SET_SSL,c,(char *)ssl)

This macro sets the ssl member of the BIO object as you can see in bio_ssl.c:

    case BIO_C_SET_SSL:
            if (ssl != NULL)
                    ssl_free(b);
            b->shutdown=(int)num;
            ssl=(SSL *)ptr;
            ((BIO_SSL *)b->ptr)->ssl=ssl;
            bio=SSL_get_rbio(ssl);
            if (bio != NULL)
                    {
                    if (b->next_bio != NULL)
                            BIO_push(bio,b->next_bio);
                    b->next_bio=bio;
                    CRYPTO_add(&bio->references,1,CRYPTO_LOCK_BIO);
                    }
            b->init=1;
            break;

The important step in this function is not the BIO_push, but rather is where it sets the ssl pointer in the BIO_SSL object to your active SSL context, i.e., ((BIO_SSL *)b->ptr)->ssl=ssl;.