I made a server with SSL and blocking sockets.
When I connect with telnet (so it does not do the handshake), the SSL_accept blocks indefinitely and blocks every new handshake/accept (and by definition new connections).
How can I solve this awful problem ?
Why not just set the socket stream to non-blocking mode before calling SSL_accept(), and then block on something like select() with a timeout if SSL_accept() returns SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE? Alternatively, you can block on select() before calling SSL_accept(). Either should work. That way you can at least bound the time the connection is blocked due to the DoS like behavior/attack.
Bear in mind that SSL/TLS is record-oriented, meaning you must loop until the full record is read. SSL_pending() can help in such cases.