Question

I’m trying to use memcache for the first time on App Engine and am hitting a “PicklingError”.

The first place I’ve tried memcache is on the site’s home page where I grab content from the datastore:

def get(self):
    content = memcache.get('home:content')
    if content is None:
        all_content = Content.all()
        all_content.order("-views")
        all_content.filter('published =', True)
        content = all_content.run(batch_size=5, limit=5)
        memcache.add(key='home:content', value=content, time=120)

(Note that this works fine without me trying to put the content Query object into memcache. Here’s the error it hits on the last line (memcache.add…):

PicklingError: Pickling of datastore_query.Batcher is unsupported.

Here’s the model for Content:

class Content(db.Model):
    category = db.StringProperty(required = True)
    content_type = db.StringProperty(required = True)
    published = db.BooleanProperty(default = False)
    title = db.StringProperty(required = True)
    abstract = db.TextProperty(required = True)
    summary = db.TextProperty(required = True)
    URL = db.LinkProperty(required = True)
    youtube_id = db.StringProperty(required = False)
    thumbnail = db.LinkProperty(required = True)
    post_author = db.StringProperty(required = True)
    author_url = db.LinkProperty(required = False)
    date_post = db.DateTimeProperty(required = True, auto_now_add = True)
    date_source = db.DateTimeProperty(required = False)    
    # todo: split out to use decent shardedcounter approach
    views = db.IntegerProperty(default = 0)
    up_votes = db.IntegerProperty(default = 0)
    down_votes = db.IntegerProperty(default = 0)    
    def votes(self):
        return self.up_votes - self.down_votes

I’m struggling to figure out what a PicklingError is and how that might relate to trying to store a Query object into memcache. My questions:
What am I doing wrong?
Is this because I’m trying to cache the iterator?
Is there any value in caching the Query object and needing to call .run() on it every page load?

Answer 1

Have a look at the source for memcache.

In a nutshell, this occurs because your value has to be serialized in a simple way, so by default pickle (actually cPickle) is used to serialize the object you pass in.

When add is called, _set_with_policy is called which subsequently calls _set_multi_async_with_policy. In _set_multi_async_with_policy, the key-value pairs are passing in as mapping and are serialized in a loop:

for key, value in mapping.iteritems():
  server_key = _key_string(key, key_prefix, user_key)
  stored_value, flags = _validate_encode_value(value, self._do_pickle)

In the helper method _validate_encode_value, if the object passed in is not something recognizable such as int, bool, str, the method attempts to pickle the object:

else:
  stored_value = do_pickle(value)
  flags |= TYPE_PICKLED

UPDATE: When you call run you get an iterator object back which contains certain objects contained in your query. If you’d like to pickle just the results, you can just cast this iterator to a list via

content = list(all_content.run(batch_size=5, limit=5))

If you’d like the other pieces kept around, you’ll need some sort of custom pickler. As you can see in Batcher:

  def __getstate__(self):
    raise pickle.PicklingError(
        'Pickling of datastore_query.Batch is unsupported.')

most of the classes defined in datastore_query — the very same classes that define the majority of the query behavior — strongly discourage pickling by throwing a PicklingError when __getstate__ is called. If you’ve never played around with it, __getstate__ and __setstate__ are custom methods that help to pickle and unpickle objects.