Question

I am building a web application on google app engine using python and jinja2. I have a website where users can write posts and I have 15 main categories and each of those has 4 divisions. Now I want to implement memcache because I have a 20:1 reader to poster ratio but how can I do it without making 60 different keys? Should I just do it that way? Or should I hit the database and sort the results and have some parameters on the function that gets those results like so:

def posts_cache(update = False, category = None, sport = None):
        key = 'main'
        posts = memcache.get(key)
        if posts is None or update:
                logging.error("DB QUERY")
                posts = db.GqlQuery("SELECT * "
                                        "FROM Post "
                                        "ORDER BY created DESC "
                                        "LIMIT 100",
                                         key)
                posts = list(posts)
                memcache.set(key, posts)
        if category and sport:
            sportcatlist = []
            for post in posts:
                     if post.category == category:
                         if post.sport == sport:
                             sportcatlist.append(post)
            return sportcatlist
        elif category:
            categorylist = []
            for post in posts:
                     if post.category == category:
                            categorylist.append(post)
            return categorylist
        elif sport:
            sportlist = []
            for post in posts:
                     if post.sport == sport:
                            sportlist.append(post)
            return sportlist
        return posts

Or is there a more efficient way to do it?

Answer 1

One possible way to do this more efficiently would be to have list of posts per category and content of the post cached separately by predefined key format (‘category_%s(category_name)’ for categories and ‘post_%s(post_key)’. First contains list posts’ keys (possibly with some meta-information like last-updated-date if needed, the second – content of the post by key). Under ‘key’ I mean either serialized datastore key or id in datastore or something else you could use to simply read post from datastore if it’s not in memcache. Storing each particular post and categories content separately will be efficient even under huge load including updates as for updated post you invalidate single memcache key, by adding/deleting post you invalidate list for single category, all other memcached data still in place so other requests handled by memcache only. As appengine enforce limits on amount of data can be stored in memcache, it will delete older and rare hit items and keep often-use which is exactly what you need and scales perfectly fine, approach you described will not give you this.
Hope it helps.