Question

I’m trying to decrypt a DES-encrypted file from an external source with a known key and IV using OpenSSL (other libraries aren’t really an option, as the application links against it already and I don’t want to introduce new dependencies). It’s a Qt application, so the data’s coming in and out as QByteArrays.

Here’s the code I have at the moment (set to write the decrypted data out to a file for inspection):

AmzHandler::AmzHandler(QByteArray encoded)
{
    QByteArray encrypted = QByteArray::fromBase64(encoded);
    QByteArray decrypted = decrypt(encrypted);
    QFile fred ("decrypted");
    fred.open(QFile::WriteOnly);
    fred.write(decrypted);
    fred.close();
}

QByteArray AmzHandler::decrypt(QByteArray encrypted)
{
    DES_cblock key = {0x29, 0xab, 0x9d, 0x18, 0xb2, 0x44, 0x9e, 0x31};
    DES_cblock iv = {0x5e, 0x72, 0xd7, 0x9a, 0x11, 0xb3, 0x4f, 0xee};
    DES_key_schedule schedule;
    unsigned char decrypted[encrypted.size()];

    DES_set_odd_parity(&key);
    DES_set_key_checked(&key, &schedule);

    DES_ncbc_encrypt((unsigned char * )encrypted.constData(), (unsigned char * )decrypted, encrypted.size(), &schedule, &iv, DES_DECRYPT);

    return QByteArray::fromRawData((char * )decrypted, length);
}

The output file for my test input is nonsense, and is not consistent across multiple runs. (I have a working implementation in Python, which is attached at the bottom of this post, to test against.) I’m not really sure what’s going on; whether I’ve made some simple screwup in the conversions to char or am misusing OpenSSL.

EDIT: Solved. It turns out the line DES_set_odd_parity(&key); was missing. Add that and it works.

Here’s working code in Python:

def AmzHandler(encoded):
    encrypted = base64.b64decode(encoded)
    d = pyDes.des(hex_to_str("29AB9D18B2449E31"), mode=pyDes.CBC, IV=hex_to_str("5E72D79A11B34FEE"))
    decrypted = d.decrypt(encrypted)
    f = open("decrypted-py", "w")
    f.write(decrypted)
    f.close()

Answer 1

The issue was just a missing call to DES_set_odd_parity(&key);, which appears to be necessary for correct decoding with this implementation. Working code is as follows:

QByteArray AmzHandler::decrypt(QByteArray encrypted)
{
    DES_cblock key = {0x29, 0xab, 0x9d, 0x18, 0xb2, 0x44, 0x9e, 0x31};
    DES_cblock iv = {0x5e, 0x72, 0xd7, 0x9a, 0x11, 0xb3, 0x4f, 0xee};
    DES_key_schedule schedule;
    unsigned char decrypted[encrypted.size()];

    DES_set_odd_parity(&key);
    DES_set_key_checked(&key, &schedule);

    DES_ncbc_encrypt((unsigned char * )encrypted.constData(), (unsigned char * )decrypted, encrypted.size(), &schedule, &iv, DES_DECRYPT);

    return QByteArray::fromRawData((char * )decrypted, length);
}