Question

I am trying to encrypt in C++ using CryptoAPI and decrypt Java using SunJCE. I have gotten the RSA key to work — and verified on a test string. However, my AES key is not working — I get javax.crypto.BadPaddingException: Given final block not properly padded.

C++ Encryption:

// init and gen key
HCRYPTPROV provider;
CryptAcquireContext(&provider, NULL, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, CRYPT_VERIFYCONTEXT);

// Use symmetric key encryption
HCRYPTKEY sessionKey;
DWORD exportKeyLen;
CryptGenKey(provider, CALG_AES_128, CRYPT_EXPORTABLE, &sessionKey);

// Export key
BYTE exportKey[1024];
CryptExportKey(sessionKey, NULL, PLAINTEXTKEYBLOB, 0, exportKey, &exportKeyLen);

// skip PLAINTEXTKEYBLOB header
//      { uint8_t bType, uint8_t version, uint16_t reserved, uint32_t aiKey, uint32_t keySize }
DWORD keySize =  *((DWORD*)(exportKey + 8));
BYTE * rawKey = exportKey + 12;

// reverse bytes for java
for (unsigned i=0; i<keySize/2; i++) {
    BYTE temp = rawKey[i];
    rawKey[i] = rawKey[keySize-i-1];
    rawKey[keySize-i-1] = temp;
}

// Encrypt message
BYTE encryptedMessage[1024];
const char * message = "Decryption Works";
BYTE messageLen = (BYTE)strlen(message);
memcpy(encryptedMessage, message, messageLen);
DWORD encryptedMessageLen = messageLen;
CryptEncrypt(sessionKey, NULL, TRUE, 0, encryptedMessage, &encryptedMessageLen, sizeof(encryptedMessage));

// reverse bytes for java
for (unsigned i=0; i<encryptedMessageLen/2; i++) {
    BYTE temp = encryptedMessage[i];
    encryptedMessage[i] = encryptedMessage[encryptedMessageLen - i - 1];
    encryptedMessage[encryptedMessageLen - i - 1] = temp;
}

BYTE byteEncryptedMessageLen = (BYTE)encryptedMessageLen;
FILE * f = fopen("test.aes", "wb");
fwrite(rawKey, 1, keySize, f);
fwrite(&byteEncryptedMessageLen, 1, sizeof(byteEncryptedMessageLen), f);
fwrite(encryptedMessage, 1, encryptedMessageLen, f);
fclose(f);

// destroy session key
CryptDestroyKey(sessionKey);
CryptReleaseContext(provider, 0);

Java Decryption:

try
{
    FileInputStream in = new FileInputStream("test.aes");
    DataInputStream dataIn = new DataInputStream(in);

    // stream key and message
    byte[] rawKey = new byte[16];
    dataIn.read(rawKey);
    byte encryptedMessageLen = dataIn.readByte();
    byte[] encryptedMessage = new byte[encryptedMessageLen];
    dataIn.read(encryptedMessage);

    // use CBC/PKCS5PADDING, with 0 IV -- default for Microsoft Base Cryptographic Provider
    SecretKeySpec sessionKey = new SecretKeySpec(rawKey, "AES");
    Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
    cipher.init(Cipher.DECRYPT_MODE, sessionKey, new IvParameterSpec(new byte[16]));

    cipher.doFinal(encryptedMessage);
}
catch (Exception e) {
  e.printStackTrace();
}

In a similar example I have tried permutations of not reversing the bytes of the key and not reversing bytes in the message. If I encrypt and decrypt with the imported key in java, I get valid results. I can also encrypt and decrypt exclusively in C++.

Questions:

1. Should I use CBC/PKCS5PADDING? Is this the default for MS_ENH_RSA_AES_PROV?
2. Is a zeroed IV indeed the default for MS_ENH_RSA_AES_PROV?
3. Are there any ways to diagnose the specifics of how the key is behaving?
4. I’d like to stick with standard Java packages instead of installing BouncyCastle, but are there any differences that would make a 3rd party package work better?

Answer 1

I had to do several things to get the message correctly:

1. Explicitly set KP_MODE to CRYPT_MODE_CBC, and KP_IV to 0
2. Use NoPadding in Java decryption
3. Don’t reverse the bytes for the key or the message

In terms of diagnosing the problem the most useful piece of advice was to set NoPadding in Java which prevents the BadPaddingException. This allowed me to see the results — even if wrong.

Strangely, the RSA Java/CryptoAPI interop solution requires the message to be completely byte reversed in order to work with Java, but AES does not expect the key or the message to be byte reversed.

CryptSetKeyParam would not let me use ZERO_PADDING, but when looking at the decrypted bytes, it is clear that CryptoAPI fills with the number of unused bytes. For instance, with a block size of 16, if the last block only uses 9 bytes, then the remaining 5 bytes get the value of 0x05. Does this present a potential security leak? Should I pad all other bytes with random bytes and use only the last byte to signify how much padding is used?

The working code (using the CryptoAPI convention of last byte being pad count) is below (checking of return values from Crypt have been removed for simplicity):

// init and gen key
HCRYPTPROV provider;
CryptAcquireContext(&provider, NULL, MS_ENH_RSA_AES_PROV, PROV_RSA_AES, CRYPT_VERIFYCONTEXT);

// Use symmetric key encryption
HCRYPTKEY sessionKey;
DWORD exportKeyLen;
BYTE iv[32];
memset(iv, 0, sizeof(iv));
DWORD padding = PKCS5_PADDING;
DWORD mode = CRYPT_MODE_CBC;
CryptGenKey(provider, CALG_AES_128, CRYPT_EXPORTABLE, &sessionKey);
CryptSetKeyParam(sessionKey, KP_IV, iv, 0);
CryptSetKeyParam(sessionKey, KP_PADDING, (BYTE*)&padding, 0);
CryptSetKeyParam(sessionKey, KP_MODE, (BYTE*)&mode, 0);

// Export key
BYTE exportKey[1024];
CryptExportKey(sessionKey, NULL, PLAINTEXTKEYBLOB, 0, exportKey, &exportKeyLen);

// skip PLAINTEXTKEYBLOB header
//      { uint8_t bType, uint8_t version, uint16_t reserved, uint32_t aiKey, uint32_t keySize }
DWORD keySize =  *((DWORD*)(exportKey + 8));
BYTE * rawKey = exportKey + 12;

// Encrypt message
BYTE encryptedMessage[1024];
const char * message = "Decryption Works -- using multiple blocks";
BYTE messageLen = (BYTE)strlen(message);
memcpy(encryptedMessage, message, messageLen);
DWORD encryptedMessageLen = messageLen;
CryptEncrypt(sessionKey, NULL, TRUE, 0, encryptedMessage, &encryptedMessageLen, sizeof(encryptedMessage));

BYTE byteEncryptedMessageLen = (BYTE)encryptedMessageLen;
FILE * f = fopen("test.aes", "wb");
fwrite(rawKey, 1, keySize, f);
fwrite(&byteEncryptedMessageLen, 1, sizeof(byteEncryptedMessageLen), f);
fwrite(encryptedMessage, 1, encryptedMessageLen, f);
fclose(f);

// destroy session key
CryptDestroyKey(sessionKey);
CryptReleaseContext(provider, 0);

Java Decryption:

try
{
    FileInputStream in = new FileInputStream("test.aes");
    DataInputStream dataIn = new DataInputStream(in);

    // stream key and message
    byte[] rawKey = new byte[16];
    dataIn.read(rawKey);
    byte encryptedMessageLen = dataIn.readByte();
    byte[] encryptedMessage = new byte[encryptedMessageLen];
    dataIn.read(encryptedMessage);

    // use CBC/NoPadding, with 0 IV -- (each message is creating it's own session key, so zero IV is ok)
    SecretKeySpec sessionKey = new SecretKeySpec(rawKey, "AES");
    Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
    cipher.init(Cipher.DECRYPT_MODE, sessionKey, new IvParameterSpec(new byte[16]));

    byte[] decryptedBlocks = cipher.doFinal(encryptedMessage);

    // check versus expected message
    byte[] expectedBytes = "Decryption Works -- using multiple blocks".getBytes();
    Assert.assertTrue("Incorrect Message" + new String(message), Arrays.equals(message, expectedBytes));
}
catch (Exception e) {
  e.printStackTrace();
}