Question

I need to encrypt binary files (ranging anywhere from tens of kb to a couple mb) in memory in a Rails application. For compliance reasons I cannot write the file to disk in unencrypted form. My concern is the memory consumption associated with this approach. I am wondering if there are any recommendations for a way to do this (perhaps there is a way I can stream/chunk the data) that will be less memory-intensive than trying to operate on the whole file at once?

I would prefer to use the openssl aes-256-cbc cipher, though I am open to other algorithms (for example, some kind of streaming cipher) if it’s reasonably secure and solves my memory issue. I have encryption working using aes-256-cbc on files already, so I’m really focused on the memory aspect, not how to actually do the encryption.

Are there any good options for encrypting large binary data streams?

Answer 1

I don’t see why you are concerned about memory consumption of any of the Cipher algorithms offered by Ruby?

If you use Cipher#update, then you will receive chunks of encrypted data that you could write to your output stream. AES block size is 16 bytes (128 bit, regardless of AES-128 or AES-256), so for each 16 byte you feed it, it will generate 16 bytes of encrypted output. This implies that there is no need to buffer your input and encrypt it all in a one-shot operation, you can read chunks of input and encrypt them using Cipher#update before writing them to your output:

# assuming io is the IO representing your uploaded file 
# and out is the IO you are writing to
while chunk = io.read(1024)
  out << cipher.update(chunk)
end
out << cipher.final