Question

I’m trying to get OpenSSL working with Java and Native C for my Android application.

What I did so far:

Initialised OpenSSL like:

ret = SSL_library_init();
SSL_load_error_strings();
ctx = SSL_CTX_new(SSLv23_method());
ret = SSL_CTX_use_certificate(ctx, sc_cert); // sc_cert is the Smart Cards auth certificate -> this is working!
_ssl = SSL_new(ctx);

Now i tryed to set the rsa_sign function (my own one) callback:

RSA_METHOD *rsameth = RSA_get_default_method();
rsameth -> rsa_verify = &sc_rsa_verify; // Just returns 1, but gets never called.
rsameth -> rsa_sign = &sc_rsa_sign;
rsameth -> flags |= RSA_FLAG_SIGN_VER; // If i would use 0x1FF my function gets called, why?
RSA_set_default_method(rsameth);

_rsa = RSA_new(); // handle error
// No need to do this: RSA_set_default_method already did that!
//_rsa -> meth = rsameth;
//_rsa -> flags |= RSA_FLAG_SIGN_VER;
//RSA_set_method(_rsa, rsameth);
ret = SSL_use_RSAPrivateKey(_ssl, _rsa);
RSA_set_default_method(rsameth);

Now the my last steps:

sbio = BIO_new_socket(sock, BIO_NOCLOSE); // Sock had been created before and is working!
SSL_set_bio(_ssl, sbio, sbio);
if(_session) SSL_set_session(_ssl, _session);
ret = SSL_connect(_ssl);

Now after SSL_connect I get either:

• No error: when my own RSA_sign (sc_rsa_sign) was NOT called
• Or: error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error, when my own RSA_sign (sc_rsa_sign) WAS called

Now you can take a look inside my own RSA_sign (sc_rsa_sign) function:

jbyteArray to_crypt = (*_env) -> NewByteArray(_env, m_length);
(*_env) -> SetByteArrayRegion(_env, to_crypt, 0, m_length, m);

// Jump into Java and do the crypt on card. This is working!
jbyteArray crypted = (*_env) -> CallObjectMethod(_env, _obj, _callback_cryptoncard, to_crypt);

// I also read that siglen should be the size of RSA_size(rsa), thus rsa -> n is not allowed to be NULL here. But it is! What is wrong here?
//int size = RSA_size(rsa);
//sigret = malloc(size);

// Obtain bytes from Java. Working (right size and crypted)!
*siglen = (*_env) -> GetArrayLength(_env, crypted);
sigret = (*_env) -> GetByteArrayElements(_env, crypted, NULL);

//(*_env) -> ReleaseByteArrayElements(_env, crypted, sigret, 0);

return 1;

Thats all I did so far. Been struggling with this for weeks now! Hope that somebody can help me!

Answer 1

I got the mistake (embarassing):

sigret = (*_env) -> GetByteArrayElements(_env, crypted, NULL);

overwrote the pointer, I changed it to:

unsigned char *sigrettemp = (*_env) -> GetByteArrayElements(_env, crypted, NULL);
memcpy(sigret, sigrettemp, siglen);

and everything is working fine now!