Question

A file format commonly used in our system is base64 encoded pickles – at the moment I can translate to and from strings in this trivial format with some simple code like this:

def dumps( objinput ):
    """
    Return an encoded cPickle
    """
    return cpickle_dumps( objinput ).encode( ENCODING )

def loads( strinput ):
    """
    Return an object from an encoded cpickle
    """
    return cpickle_loads( strinput.decode( ENCODING ) )

I’d like to implement a new function, called “load” which works much like the standard pickle load function except that it will read just enough bytes from a stream and then return the decoded object. The trick here is to read just enough bytes – the cPickle.load function does this, it pops bytes off the stream until the pickled data is complete. My new function needs to look like this:

def load( stream_input ):
    """"
    Return just one object popped from the stream. 
    If the stream has ended, raise an exception
    Do not over-read the stream.

How might I do this when the data is base64 encoded pickle? The difficulty here seems to be that it’s not obvious that this can be done either by wrapping the cPickle.load function or by wrapping the input stream, an additional difficulty is that what may be one-byte of pickle data may be encoded to more than one byte of base64 so it’s not clear how to adapt stream_input.read(n) into something which is guaranteed to return n bytes of decoded data when the encoded input stream is base64.

Answer 1

I think you can in fact do this by implementing a file-like object, which can then be passed directly to cPickle.load

class Base64PickleAdapter:
  def __init__(self, base64_input):
    ...

  def read(size=-1):
    ...   

  def readline():
    ... # Can be implemented in terms of read.

cPickle only requires read and readline. You don’t need to return size bytes from Base64PickleAdapter.read. The read contract allow you to return less, which should simplify implementation.