Question

I am trying to write a simple RSA encryption scheme to encrypt a small message between a single server and potentially many clients.

The plan I have come up with is to have the client generate a key pair and send the public key to the server. The server has an object that will store the socket and the public key. When it comes time to send the message the server will call a method in the Client object that will encrypt the message and send it down to the client.

I will only be encrypting the messages going to the client, so I don’t need to worry about a full transaction of keys.

I think I have that pretty well figured out.

The problems are coming from trying to connect two different implementations of RSA. The server is written in C# and the client is a MetaTrader client using C. I have decided to use OpenSSL to try to bridge the gap, but so far I haven’t been able to produce a key on the client. I am using the openssl wrapper for c# (http://openssl-net.sourceforge.net/) and I am trying to write a very basic wrapper dll for MetaTrader. So far it contains two functions, GenerateRSAKeys and DecryptRSA.

For now, I would just like to produce a PEM formatted string that contains the public key. This is what I have so far for GenerateRSAKeys().

extern "C" unsigned char *__stdcall GenerateRSAKeys (){

int len;
unsigned char *buf = new unsigned char[1024];

rsa = RSA_generate_key(512,RSA_F4,NULL,NULL);

len=i2d_RSAPublicKey(rsa, &buf);

return buf;}

Right now, when sending buf to the server I receive a single character. Such as, “X”

So my first question is, what is the best way to export the generated public key?

Answer 1

You are returning a pointer to a buffer allocated on the stack; this buffer will be overwritten when the function returns. Try buf = new unsigned char[1024];