Question

Which function in OpenSSL c library gets the certificate from the server, checks whether a valid certificate and returns according to the validity. Like valid, untrusted etc?

Answer 1

After establishing the connection just call SSL_get_verify_result(ssl) on your connection object ssl

if (SSL_get_verify_result(ssl) != X509_V_OK) {
       /* verification failed */
}

the possible return-codes for SSl_get_verify_result() can be found at the man page in the Diagnostics section

if you need to get the certificate and validate it yourself, you might want to look at
SSL_get_peer_certificate(ssl) to get the certificate and X509_verify_cert().