Question

I am trying to encrypt some plain text with a public key using RSA_public_encrypt(), this data would then be sent to a remote server for validation. I believe I have the encryption/decryption working, since the output of RSA_public_encrypt can be passed to RSA_private_decrypt and it works. The problem I am having now is I need to base64 encode the data in order to send it over HTTP.

As a test (before sending it to the server) I am encoding to base64 the output of RSA_public_encrypt(), then decoding it and passing it back into RSA_private_decrypt(). This appears to work some of the time, and fails with an error like this:

error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error

When I use memcmp to compare the original data (pre-base64) to the output of my base64 decode function I receive a -1 despite the contents appearing to match (in Visual Studio by viewing the contents of the memory as hex). I have also checked the base64 encoded version with various online tools and they appear to decode to the expected value.

I have double checked that the input/output from the base64/unbase64 functions are null terminated which appears to make little difference.

I’ve been going round in circles with this problem for a couple of days but I believe it must be something with the base64 encode/decode process because when that is not involved everything works. If anyone has any advice on how this could be happening it would be appreciated.

Openssl version: 1.0.1c

Platform: Windows/MSVC

Base64 Code:

char *base64(const unsigned char *input, int length)
{
  BIO *bmem, *b64;
  BUF_MEM *bptr;
  char *buff = NULL;

  b64 = BIO_new(BIO_f_base64());
  BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL);
  bmem = BIO_new(BIO_s_mem());
  b64 = BIO_push(b64, bmem);
  BIO_write(b64, input, length);
  BIO_flush(b64);
  BIO_get_mem_ptr(b64, &bptr);

  buff = (char *)malloc(bptr->length+1);
  memcpy(buff, bptr->data, bptr->length);
  buff[bptr->length] = '\0';

  BIO_free_all(b64);

  return buff;
}

Unbase64 Code:

char *unbase64(unsigned char *input, int length)
{
  BIO *b64, *bmem;

  char *buffer = (char *)malloc(length+1);
  memset(buffer, 0, length+1);

  b64 = BIO_new(BIO_f_base64());
  BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL);
  bmem = BIO_new_mem_buf(input, length);
  bmem = BIO_push(b64, bmem);

  BIO_read(bmem, buffer, length);
  buffer[length] = '\0';

  BIO_free_all(bmem);

  return buffer;
}

Encrypted “hello world” example:

š:Œ¼JŒ"ÌïëŸÔè#¢Oo‚À–    œê\çrú¿±a/8ƒòÌ¢Q\T¹]nío

Base64 version (using the above code):

G5qdOgWMvEqMIswZ7+uf1OgPI6JPb4LAlgmc6lzncvq/sWEvOIPyzByiUVwMjYFUuV0Vbu1v

Thanks for any help!

Answer 1

You are passing in the correct size in the unbase64 function? It should be the size of the base64 buffer returned, not the size of the destination buffer i.e. using an example main function:

int main(void)
{
  unsigned char bufron[2000];
  int i;
  char *chab;
  unsigned char *chac;

  for (i = 0; i < 2000; i++) {
      bufron[i] = i % 255;
  }

  chab = base64(bufron, 2000);
  printf("%s\n", chab);

  chac = unbase64(chab, strlen(chab));

  for (i = 0; i < 2000; i++) {
      if (bufron[i] != chac[i]) {
          printf("Failed at %d\n", i);
          return (1);
      }
  }
}