Question

I’m trying to learn how to use the OpenSSL library (total newb) and am having a bit of trouble figuring out how to have a client connect to a server with the correct cert, and fail if the cert is incorrect. My use case is build a p2p application with no central CA, so I cannot rely on a CA cert. Specifically, the server has a cert/key as usual, and the client will determine a server’s cert by asking other p2p nodes to vote.

I have two specific questions:

• In the below code snippets, I expect the client to not connect if I
  comment out the selection below ‘cipher list’. It still works! I’m missing something, right?
• Is there a way to just have the client use a server cert, and have that be sufficient for opening up a connection? Ie no key, no CA?

client:

ctx = SSL_CTX_new(DTLSv1_client_method());
SSL_CTX_set_cipher_list(ctx, "HIGH:!DSS:!aNULL@STRENGTH");

// If I comment out below stuff, client still connects happily!?
if (!SSL_CTX_use_certificate_file(ctx, "certs/server-cert.pem", SSL_FILETYPE_PEM))
    printf("\nERROR: no certificate found!");

if (!SSL_CTX_use_PrivateKey_file(ctx, "certs/server-key.pem", SSL_FILETYPE_PEM))
    printf("\nERROR: no private key found!");

if (!SSL_CTX_check_private_key (ctx))
    printf("\nERROR: invalid private key!");

server:

SSL_CTX_set_cipher_list(ctx, "HIGH:!DSS:!aNULL@STRENGTH"); // high strength ciphers
SSL_CTX_set_session_cache_mode(ctx, SSL_SESS_CACHE_OFF);

if (!SSL_CTX_use_certificate_file(ctx, "certs/server-cert.pem", SSL_FILETYPE_PEM))
    printf("\nERROR: no certificate found!");

if (!SSL_CTX_use_PrivateKey_file(ctx, "certs/server-key.pem", SSL_FILETYPE_PEM))
    printf("\nERROR: no private key found!");

if (!SSL_CTX_check_private_key (ctx))
    printf("\nERROR: invalid private key!");

My code is at https://github.com/a34729t/exp/tree/master/tun2udp/dtls; it’s built of Robin Seggelmann’s DTLS examples. Specifically, I’m working with server3_oo.c.

Answer 1

SSL/TLS can operate in different modes. The most common mode is server authentication only, in which only the server has a certificate and key. Then there is mutual authentication mode, also called client authentication, in which the client also has a (client) certificate and key. And then there is a fully anonymous mode where neither server nor client authenticate themselves, and neither needs a certificate and key.

Unless you tell OpenSSL otherwise, it will operate in server authentication only mode. In this mode, these lines here in the client:

if (!SSL_CTX_use_certificate_file(ctx, "certs/server-cert.pem", SSL_FILETYPE_PEM))
    printf("\nERROR: no certificate found!");
if (!SSL_CTX_use_PrivateKey_file(ctx, "certs/server-key.pem", SSL_FILETYPE_PEM))
    printf("\nERROR: no private key found!");

will cause the client to load a client certificate (btw, you do never have to load the server cert and key in the client). However, since the server does not ask the client for a certificate, the client never presents one. Commenting these lines out makes the client not load any client certs, which makes no difference since they aren’t being used anyway.

To make the server ask the client for a certificate, and have the server refuse the connection if none is presented, you have to tell the server like this (error handling omitted):

SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);

SSL_VERIFY_PEER makes the server ask for a client cert, and SSL_VERIFY_FAIL_IF_NO_PEER_CERT causes the server to abort the connection if the client presents no certificate (see documentation). If you don’t use the latter option, you can check for yourself if the client sent a certificate using the SSL_get_peer_certificate function.