Question

Well, I’ve been going through my personal hell these days

I am having some trouble decrypting a message that was encrypted using
RSA and I’m always failing with a “RSA/OAEP-MGF1(SHA-1): invalid
ciphertext”

1. I have a private key encoded in base64 and I load it:

       RSA::PrivateKey private_key;
       StringSource file_pk(PK,true,new Base64Decoder);
       private_key.Load( file_pk );
   

2. I then proceed to decode the message by doing:

   RSAES_OAEP_SHA_Decryptor decryptor(private_key);
   
   AutoSeededRandomPool rng;
   
   string result;
   StringSource(ciphertext, true,
       new PK_DecryptorFilter(rng, decryptor,
           new StringSink(result)
       )
   );
   

As far as I can tell, the message should be being parsed without any
problems. ciphertext is an std::string, so no \0 at the end that could
do something unexpected.

I just though of something, and what if the private key is incorrect
but can be loaded anyway without throwing a BER decode error. What
would that throw when decrypting?

Hope that anyone can shed some light on this.

Cheers

Answer 1

If the key was actually corrupted, the Load function should have failed. However you can ask the key to self-test itself, which should detect any corruption, by calling Validate, like:

bool key_ok = private_key.Validate(rng, 3);

The second parameter (here, 3) specifies how much checking to be done. For RSA, this will cause it to run all available tests, even the slow/expensive ones.

Another reason the decoding might fail is if the key simply is not the one that was used to encrypt the original message.

Obviously the ciphertext input must be completely identical to what was originally produced on the encrypting side. For debugging, one good way to check this would be to feed the ciphertext at both sides into a hash function (conveniently already available to you, of course) and comparing the outputs. If you hex or base64 encoded the ciphertext for transmission you must undo that before you give it to the RSA decryptor.