Question

I’m working on a project using Pyramid 1.3 (Python 2.7) and storing data in MySQL. I have a table of email addresses, and I would like to encrypt them for storage. I am trying to encrypt them in the application, and then will decrypt them for viewing. I’m not going for complete security but am mainly aiming to obfuscate the data enough were the database itself compromised.

I’m using PyCrypto with AES, and have been trying to follow some posts on here and some web tutorials I found. The closest I found so far is this post, and it seems to work, at least encrypting it. I follow that and get something like "7hBAQrWhJRnL9YdBGJfRErGFwGi3aC6noGzYTrGwAoQ=" stored in the database. But the decrypt function keeps erroring with this:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xa1 in position 1: ordinal not in range(128)

I came across some unicode presentation about Python which sort of helped me make more sense of it but I still keep getting the same error.

Is there a straightforward tutorial on how to encode, store in a database, pull out of database, and decode a source data string?

Do I need a specific collation on the database column? Does the field need to be a certain type? So far I’ve been using a default collation and setting it to VARCHAR, assuming that I was storing a string. It sounds like I’ve got some encoding problem somewhere with incompatible types or something but my head is spinning on where I need to change something.

Any better pointers or anything else I can provide? I can show my code but its basically a copy of the link above… I was just trying to get a proof of concept working before modifying it too much.

edit:
some sample source…
In MySQL, the table is
id (int)
client_id (int)
emailaddress varchar(100) utf8mb4_general_ci (I’ve been playing around with the collations, I have no idea what it should be!)

Python:

from base64 import b64encode, b64decode, urlsafe_b64decode, urlsafe_b64encode

BLOCK_SIZE = 32
INTERRUPT = u'\u0001'
PAD = u'\u0000'
def AddPadding(data, interrupt, pad, block_size):
    new_data = ''.join([data, interrupt])
    new_data_len = len(new_data)
    remaining_len = block_size - new_data_len
    to_pad_len = remaining_len % block_size
    pad_string = pad * to_pad_len
    return ''.join([new_data, pad_string])
def StripPadding(data, interrupt, pad):
    return data.rstrip(pad).rstrip(interrupt)#data.rsplit(interrupt,1)[0]#rstrip(pad).rstrip(interrupt)

SECRET_KEY = u'a1b2c3d4e5f6g7h8a1b2c3d4e5f6g7h8'
IV = u'12345678abcdefgh'

cipher_for_encryption = AES.new(SECRET_KEY, AES.MODE_CBC, IV)
cipher_for_decryption = AES.new(SECRET_KEY, AES.MODE_CBC, IV)

def EncryptWithAES(encrypt_cipher, plaintext_data):
    plaintext_padded = AddPadding(plaintext_data, INTERRUPT, PAD, BLOCK_SIZE)
    encrypted = encrypt_cipher.encrypt(plaintext_padded)
    return urlsafe_b64encode(encrypted)
def DecryptWithAES(decrypt_cipher, encrypted_data):
    decoded_encrypted_data = urlsafe_b64decode(encrypted_data)
    decrypted_data = decrypt_cipher.decrypt(decoded_encrypted_data)
    return StripPadding(decrypted_data, INTERRUPT, PAD)

#encrypts it
posted_singleaddress = EncryptWithAES(cipher_for_encryption, posted_singleaddress)

#"[e-mail]" inserts "Ktpr49Uzn99HZXbmqEzGKlWo9wk-XBMXGZl_iyna-8c=" into the database

clientemails is the list of emails from the table above. I get the error when uncommenting out:

#if clientemails:
#    decrypted = DecryptWithAES(cipher_for_decryption, clientemails[0].emailaddress)

I was just trying to decode the first item just to try and get it to work but that’s the part that seems to be giving it fits now….

Answer 1

The general rule with PyCrypto is that cryptographic keys, IVs, plaintexts, paddings, and ciphertexts should always be defined as binary strings, not text. The fact you use Unicode for them is by itself a source of problems.

Another problems is that you pass to AES.new key and IV in hexadecimal encoded form, so that the former is 256 bits and the latter 128 bits. That seems still to work, but I guess your intention was to use AES128 – which has a 128 bit key. You therefore need to convert it to binary, for instance via unhexlify: the two character string b’34’ will map to the single byte ‘\x34’. The IV needs to be twice as long.

In your code it’s therefore better to have:

from binascii import unhexlify

INTERRUPT = b'\x01'
PAD = b'\x00'
SECRET_KEY = unhexlify('a1b2c3d4e5f6g7h8a1b2c3d4e5f6g7h8')
IV = unhexlify('12345678abcdefgh'*2)

If you need to encrypt text, you would first encode it (e.g. to UTF-8) and then pass it to your function EncryptWithAES(). See also this example taken from the PyCrypto API:

from Crypto.Cipher import AES
from Crypto import Random

key = b'Sixteen byte key'
iv = Random.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CFB, iv)
msg = iv + cipher.encrypt(b'Attack at dawn')

The result of the encryption step (that is, the ciphertext) is again a binary string. In order to store it directly in the MySQL DB you must use either a BINARY or a VARBINARY type column.