Question

Recently, the StackOverflow community helped me develop a fairly concise @memoize decorator that is able to decorate not only functions but also methods and classes in a general way, ie, without having any foreknowledge of what type of thing it will be decorating.

One of the problems that I ran into is that if you decorated a class with @memoize, and then tried to decorate one of it’s methods with @staticmethod, this would not work as expected, ie, you would not be able to call ClassName.thestaticmethod() at all. The original solution that I came up with looked like this:

def memoize(obj):
    """General-purpose cache for classes, methods, and functions."""
    cache = obj.cache = {}

    def memoizer(*args, **kwargs):
        """Do cache lookups and populate the cache in the case of misses."""
        key = args[0] if len(args) is 1 else args
        if key not in cache:
            cache[key] = obj(*args, **kwargs)
        return cache[key]

    # Make the memoizer func masquerade as the object we are memoizing.
    # This makes class attributes and static methods behave as expected.
    for k, v in obj.__dict__.items():
        memoizer.__dict__[k] = v.__func__ if type(v) is staticmethod else v
    return memoizer

But then I learned about functools.wraps, which is intended to make the decorator function masquerade as the decorated function in a much cleaner and more complete way, and indeed I adopted it like this:

def memoize(obj):
    """General-purpose cache for class instantiations, methods, and functions."""
    cache = obj.cache = {}

    @functools.wraps(obj)
    def memoizer(*args, **kwargs):
        """Do cache lookups and populate the cache in the case of misses."""
        key = args[0] if len(args) is 1 else args
        if key not in cache:
            cache[key] = obj(*args, **kwargs)
        return cache[key]
    return memoizer

Although this looks very nice, functools.wraps provides absolutely no support for either staticmethods or classmethods. Eg, if you tried something like this:

@memoize
class Flub:
    def __init__(self, foo):
        """It is an error to have more than one instance per foo."""
        self.foo = foo

    @staticmethod
    def do_for_all():
        """Have some effect on all instances of Flub."""
        for flub in Flub.cache.values():
            print flub.foo
Flub('alpha') is Flub('alpha')  #=> True
Flub('beta') is Flub('beta')    #=> True
Flub.do_for_all()               #=> 'alpha'
                                #   'beta'

This would work with the first implementation of @memoize I listed, but would raise TypeError: 'staticmethod' object is not callable with the second.

I really, really wanted to solve this just using functools.wraps without having to bring back that __dict__ ugliness, so I actually reimplemented my own staticmethod in pure Python, which looked like this:

class staticmethod(object):
    """Make @staticmethods play nice with @memoize."""

    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        """Provide the expected behavior inside memoized classes."""
        return self.func(*args, **kwargs)

    def __get__(self, obj, objtype=None):
        """Re-implement the standard behavior for non-memoized classes."""
        return self.func

And this, as far as I can tell, works perfectly alongside the second @memoize implementation that I list above.

So, my question is: Why doesn’t the standard builtin staticmethod behave properly on it’s own, and/or why doesn’t functools.wraps anticipate this situation and solve it for me?

Is this a bug in Python? Or in functools.wraps?

What are the caveats of overriding the builtin staticmethod? Like I say, it seems to be working fine now, but I’m afraid that there might be some hidden incompatibility between my implementation and the builtin implementation, which might blow up later on.

Thanks.

Edit to clarify: In my application, I have a function that does an expensive lookup, and gets called frequently, so I memoized it. That is quite straightforward. In addition to that, I have a number of classes that represent files, and having multiple instances representing the same file in the filesystem will generally result in inconsistent state, so it’s important to enforce only one instance per filename. It’s essentially trivial to adapt the @memoize decorator to this purpose and still retain it’s functionality as a traditional memoizer.

Real world examples of the three different uses of @memoize are here:

• On a function
• On a method
• On a class containing staticmethods

Answer 1

Several thoughts for you:

• The operation of a staticmethod is completely orthogonal to the operator of class decorators. Making a function into a staticmethod only affect what happens during attribute lookup. A class decorator is a compile-time transformation on a class.

• There isn’t a “bug” in functools.wraps. All it does is copy function attributes from one function to another.

• As currently written, your memoize tool does not account for the different call signatures for classmethods and staticmethods. That is a weakness in memoize not in the class tools themselves.

I think you’ve imagined tools like class decorators, staticmethods, classmethods, and functools to have some sort of mutually integrating intelligence. Instead, all of these tools are very simple and need the programmer to consciously design their interactions.

ISTM that the underlying problem is that the stated goal is somewhat underspecified: “decorator that is able to decorate not only functions but also methods and classes in a general way, ie, without having any foreknowledge of what type of thing it will be decorating.”

It is not entirely clear what the semantics of memoize would be in each scenario. And there is no way for Python’s simple components to automatically compose themselves in a way that would be able guess what you really wanted to do.

My recommendation is that you start with a list of worked out example of memoize using with a variety of objects. Then start building out your current solution to get those to work one at a time. At each step, you will learn where your spec doesn’t match what meoize actually does.

Another thought is that functools.wraps and class decorators aren’t strictly necessary for this problem. Both can be implemented manually. Start with wiring your tool to do what you want it to do. Once it is working, then look at replacing steps with wraps and decorators. That beats trying to force the tools to your will in situations where they may not be a good fit.

Hope this helps.