Question

I’m trying to run openSSL examples, (source downloaded from here). I’ve managed to build it and running the ‘wserver’ example on port 4433 (running on Ubuntu 12.10 virtual machine using VirtualBox)

I then try to launch a client from Windows 8, but I keep getting SSL accept error shows on the server’s terminal.

I saw this post and copied the bash script from the accepted answer, using it, I’ve managed to create a new root.pem and other certificates, but I’m still getting the same error

(Don’t know it if matters, I’m using port 4433 because the VM couldn’t bind to 443 for some reason, and Yes, I changed it on the client as well)

Also, server written in C.

Any ideas?
Thanks!

EDIT

I found that the error happens in this line

r=SSL_accept(ssl);

I checked the error returned into r using

SSL_get_error(ssl,r);

and it appeard to be SSL_ERROR_SYSCALL

I checkd ERR_get_error(); and it returned 0
meaning

If ret == 0, an EOF was observed that violates the protocol

Not sure yet what it means..

EDIT 2

I checked the client return code, and it’s error 12045
which, according to MSDN is

ERROR_INTERNET_INVALID_CA

12045

The function is unfamiliar with the Certificate Authority that generated the server’s certificate.

But when using the supplied client from the virtual machine, the connection is fine and no certificates problems

Overriding 12045 as shown by Microsoft here, leads to a 12038 error, which is

ERROR_INTERNET_SEC_CERT_CN_INVALID

           SSL certificate common name (host name field) is incorrect.
           For example, if you entered www.server.com and the common
           name on the certificate says www.different.com.

Does this help?
Thanks again!

Answer 1

Because the error code was SSL_ERROR_SYSCALL and the r was zero, the client did close the underlying TCP connection gracefully (by calling close() or shutdown()).

You should look from the client code, what may cause the closing of TCP connection.

Edit: according to edit2 of the question:

There are some problems with certificates.

I suggest to use self-signed cerificates during testing for avoiding problems with certificate chains.
When everything works fine, then start using more complex certificates.

Because you get ERROR_INTERNET_SEC_CERT_CN_INVALID, there may be wrong common nane (CN) in a certificate. Check the Subject: field of certificate. There is CN=xxxxx in the subject. The xxxxx must be same as IP/hostname of the host.

Or you can also remove the common name check from your client code.
You have probably the following code from the example:

if(require_server_auth)
  check_cert(ssl,host);

Comment it out or set require_server_auth=0. And then try again.