Question

I’m using a private RSA key to encrypt a random AES key with the default Java RSA implementation:

Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.ENCRYPT_MODE, privateKey);
byte[] cipherBytes = cipher.doFinal(plainText.getBytes());

Since we need a public key anyway, this is a convenient method to disguise the key and make sure it had been encrypted with our private key. The decryption is done similarly:

Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.DECRYPT_MODE, publicKey);
byte[] plainBytes = cipher.doFinal(cipherBytes);

This works fine with Oracle’s JDK, but with IBM’s this fails because IBM thinks using the private key for encryption is not a valid use case. Unfortunately, I have to support both JDKs, so I’m trying to re-implement the RSA decryption myself.

This is the code I have so far:

BigInteger big = new BigInteger(cipherBytes);
big = big.modPow(pub.getPublicExponent(), pub.getModulus()); 
System.out.println(new String(big.toByteArray()));

It almost works, but there seems to be a padding issue. Most of the time I’m getting the original text with a string of dot-like symbols in front of it, but sometimes it’s only random bytes.

Unfortunately, I wasn’t able to figure out which padding scheme is used by default. Does anyone know what’s missing in my code or can at least give a hint with which algorithm the padding is handled?

Here is an example of input and output values, as requested. I have used 512 bit keys to avoid too huge numbers.

Public modulus :  8117919732251191237549784557538073836207094968952416063837701691514861428726690140363567956265691836505266266364256892197254736023284927189008247933889303
Public exponent:  65537
Plaintext:        teststring
Plaintext as BN:  549665952565679142563431
Ciphertext as BN: 6304229782339071167863563708554898540621778162930150363326921290545577949349781053660336996882823758722402137580193903457839924005473545992074817339077456
"Decrypted" BN:   409173825987017733751648712103449894027080255755383098685411421012016724550584319360408761540738019643860835515945008876151848132891805352276483731047
Resultstring: ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇteststring

To address the discussion why am I doing this:

The public key is hard-coded into my software. I use the private key to encrypt another key for AES. Therefore, to actually decode anything with AES, you need the AES key first. To get this key, you have to decrypt it with the public key first. Since the public key cannot be modified without serious manipulation, only AES keys encrypted with the private key work. You may extract the public key somehow and decrypt the AES key, but that’s elaborate and only gets you the AES key to decrypt the secured content. There is also a signature calculated with the private key, which is verified with the public key as well. So manipulations aren’t possible.

So yes, technically the signature is sufficient, because there are methods to read the content. But those are elaborate and I don’t mind if anyone really takes all the trouble, but I don’t want to make things easy.

Answer 1

Ok, I’ve figured it out by reading the RSA spec. To add more security, a padding is added before encryption and the following “string” is created:

0x00 + BT + Padding + 0x00 + Data

The block type (BT) indicates the kind of padding. With BT = 0x01 the padding is 0xff and with BT = 0x02 the padding is random but non-zero. The concatenated string is then encrypted.

When decrypting, the format can be verified, but to just read the data, the leading bytes have to be removed. They are all non-zero until the 0x00 just before the data. Therefore, everything until and including the 0x00 after the padding can be removed. What’s left is the message.

This code works now:

// Decrypt
byte[] decryptedBytes = (new BigInteger(1, cipherBytes)).modPow(pub.getPublicExponent(), pub.getModulus()).toByteArray();

// Extract msg
int dataStart;
for (dataStart = 0; decryptedBytes[msgStart] != 0; dataStart++);
dataStart++;

byte finalBytes[] = new byte[decryptedBytes.length - msgStart];
System.arraycopy(decryptedBytes, msgStart, finalBytes, 0, finalBytes.length);

This also explains the string of “^” in my previous attempts. Those were the padding bytes, which are 0xff with BT = 0x01.

I only needed decryption, but for the sake of completeness, this is the code for encryption:

int bitLength = 512;
String plainText = "teststring";

// Convert to bytes
byte plainBytes[] = plainText.getBytes(); 

byte encryptionBytes[] = new byte[bitLength / 8];

encryptionBytes[0] = 0; // Leading 0
encryptionBytes[1] = 1; // Block type

// Padding String
int paddingEnd = (bitLength / 8) - plainBytes.length - 2;
for (int i = 2; i < paddingEnd; i++) {
    encryptionBytes[i] = (byte) 0xff;
}
encryptionBytes[paddingEnd + 1] = 0;

// Actual data
System.arraycopy(plainBytes, 0, encryptionBytes, paddingEnd + 1, plainBytes.length);

// Encrypt
byte[] cipherBytes = (new BigInteger(1, encryptionBytes)).modPow(priv.getPrivateExponent(), priv.getModulus()).toByteArray();

Hope this helps anyone 🙂