Question

I am writing an SMTP server in Ruby, and I am trying to figure out what the correct way to handle the SSL/TLS error “decryption failed or bad record mac” is

Retrying the operation spits out random garbage from #read_nonblock, and apart from being rude and closing the connection, I’m not sure what to do

Answer 1

Yourself being the server here, the message would indicate that either OpenSSL wasn’t able to properly decrypt the TLS records sent by the client or that the MAC computation failed, i.e. the client’s tag doesn’t match the one you computed.

Apart from bugs or network errors (as EJP mentioned), this can also happen if someone actively changed the content sent, the wrong message was sent or only a partial message was sent etc. etc.

In any case, as the server, you don’t want to try to rescue such a connection. After all, it could be an attack. The proper way to handle this event is to call SSLSocket#close which will try to execute a SSL_shutdown which again will attempt to send the “close notify” alert to the client, which is the graceful TLS way of telling the peer that you’re about to close this connection.

In order to resume communication with a client where this happened, you would negotiate a new connection started from scratch.