Question

Anyone happen to use this combination in their web application? I’m having a bit of trouble finding some sort of tutorial or guideline for configuring this. Also seeing as how I started using Pylons recently I’m not familiar so please keep the advice very newbie friendly ( I haven’t even used modules like Beaker all that much ).

I’m using MySQL, running pastie HTTP server, just installed memcached package on Debian, using SQLAlchemy ORM to interact with my DB in my Pylons app, now I’m not sure what to do.

Answer 1

memcached is nice and framework-agnostic, and you just have to write a bit of code to interact with it. The general idea of memcached is:

object = try_memcached()
if not object:
    object = real_query()
    put_in_memcached(object)

That will likely be done in your SQLAlchemy abstraction, in your case. Since I’m unfamiliar with your entire platform (and only memcached), I did a bit of Googling.

This blogger appears to have implemented them together, and has helpfully provided a link to the code he uses. The relevant code appears to be this, which might make sense to you:

#!/usr/bin/python
"""
memcached objects for use with SQLAlchemy
"""
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.

import memcache
import sqlalchemy
import sqlalchemy.orm

SQLA_SESSION = sqlalchemy.orm.sessionmaker()

MEMCACHED_CLIENT = memcache.Client(['127.0.0.1:11211'])

class DetachedORMObject(object):
    """
    Session-detached object for use with ORM Mapping.  As the SQLAlchemy
    documentation indicates, creating and closing a session is not analogous
    to creating and closing a database connection.  Connections are pooled by
    the database engine by default.  Creating new sessions is of a minimal
    cost.  Also, objects using this wrapper will not likely interact in with
    the database through the full power of SQLAlchemy queries.

    """
    @classmethod
    def fetch_by_field(cls, field, value):
        """Fetch a mapped orm object with the give field and value"""
        session = SQLA_SESSION()
        try:
            class_object = session.query(cls).filter(field == value).one()
        except sqlalchemy.orm.exc.NoResultFound:
            class_object = None
        finally:
            session.close()
        return class_object

    def update(self):
        """Update the database with the values of the object"""
        session = SQLA_SESSION()
        session.add(self)
        session.commit()
        session.refresh(self)
        session.close()

    def refresh(self):
        """Refresh the object with the values of the database"""
        session = SQLA_SESSION()
        session.add(self)
        session.refresh(self)
        session.close()

    def delete(self):
        """Delete the object from the database"""
        session = SQLA_SESSION()
        session.add(self)
        session.delete(self)
        session.commit()
        session.close()


class MemcachedObject(object):
    """
    Object Wrapper for serializing objects in memcached. Utilizes an abstract
    method, get_isntance_key, to understand how to get and set objects that
    impliment this class.
    """
    @classmethod
    def get_cached_instance(cls, instance_key):
        """Retrieve and return the object matching the instance_key"""
        key = str(cls.__module__ + '.' + cls.__name__ + ':' \
           + str(instance_key))
        print "Memcached Getting:", key
        return MEMCACHED_CLIENT.get(key)

    def set_cached_instance(self, time=0, min_compress_len=0):
        """Set the cached instance of an object"""
        print "Memcached Setting:", self.get_cache_key()
        return MEMCACHED_CLIENT.set(self.get_cache_key(), self, time, \
            min_compress_len)

    def delete_cached_instance(self, time=0):
        """Wrapper for the memcached delete method"""
        print "Memcached Deleting:", self.get_cache_key()
        return MEMCACHED_CLIENT.delete(self.get_cache_key(), time)

    def get_cache_key(self):
        """Prepends the full class path of the object to the instance key"""
        return self.__class__.__module__ + '.' + \
            self.__class__.__name__ + ':' + self.get_instance_key()

    def get_instance_key(self):
        """Get the instance key, must be implemented by child objects"""
        raise NotImplementedError \
            ("'GetInstanceKey' method has not been defined.")


class MemcachedORMObject(DetachedORMObject, MemcachedObject):
    """
    Putting it all together now.  Implements both of the above classes. Worth
    noting is the method for checking to see if the fetch_by_field method is
    invoked using a primary key of the class.  The same technique is used to
    generate an instance key for an instance of the class.
    """
    @classmethod
    def fetch_by_field(cls, field, value):
        """Fetch the requested object from the cache and database"""
        orm_object = None
        matched_primary_key = True
        for key in cls._sa_class_manager.mapper.primary_key:
            if field.key != key.key:
                matched_primary_key = False
        if matched_primary_key:
            orm_object = cls.get_cached_instance('(' + str(value) + ')')
        if orm_object is None:
            orm_object = super(MemcachedORMObject, cls). \
                fetch_by_field(field, value)
            if orm_object is not None:
                orm_object.set_cached_instance()
        return orm_object

    def update(self):
        """Update the object in the database and memcached"""
        DetachedORMObject.update(self)
        self.set_cached_instance()

    def refresh(self):
        """Refresh the object from the database and memcached"""
        DetachedORMObject.refresh(self)
        self.set_cached_instance()

    def delete(self):
        """Delete the object from the database and memcached"""
        DetachedORMObject.delete(self)
        self.delete_cached_instance()

    def get_instance_key(self):
        """Get the instance key, implimenting abstract method in base"""
        key = []
        for column in self._sa_instance_state.manager.mapper.primary_key:
            key.append('(' + str(getattr(self, column.key)) + ')')
        return ''.join(key)

Not sure if that helps, but there you have it. You can see that memcached idiom in use:

    if matched_primary_key:
        orm_object = cls.get_cached_instance('(' + str(value) + ')')
    if orm_object is None:
        orm_object = super(MemcachedORMObject, cls). \
            fetch_by_field(field, value)