Question

I want to understand python metaclasses. For practice I’m implementing a declarative way for writing classes (similar to sqlalchemy.ext.declarative). This looks promising as long as I only have one attribute.

But when I add another attribute, some part of the first attribute is changed and the value of the first attribute is validated against the pattern of the second attribute. This might be caused by the metaclass, by a closure, by the property or a combination of them. I try to give a minimal, complete but readable example.

#! /usr/bin/env python

"""
Something like:
    class Artist:
        locale = Pattern('[A-Z]{2}-[A-Z]{2}')

should be equivalent to:
    class Artist:
        def __init__(self):
            self._locale = None
        @property
        def locale(self):
            return self._locale
        @locale.setter
        def locale(self, value):
            validate(value, '[A-Z]{2}-[A-Z]{2}')
            self._locale = value

Problem:
    The code below works if Artist has only one attribute.
    When I add another one with a different pattern, only that last
    pattern is used in validation.
"""

import re
import unittest


# this class (and future siblings) are used to describe attributes
class Pattern(object):
    def __init__(self, pattern):
        self.pattern = pattern

    def validate(self, value):
        if value is None:
            return
        if not re.match("^%s$" % self.pattern, value):
            raise ValueError("invalid value: %r" % value)

    def __repr__(self):
        return "%s(pattern=%r)" % (self.__class__.__name__, self.pattern)


# __metaclass__ based class creation
def createClassFromDeclaration(name, bases, dct):
    """ Examine dct, create initialization in __init__ and property. """
    attributes = dict()
    properties = dict()
    for key, value in dct.iteritems():
        if not isinstance(value, Pattern):
            continue
        pattern = value
        pattern.attribute = "_%s" % key
        attributes[key] = pattern

        def fget(self):
            return getattr(self, pattern.attribute)
        def fset(self, value):
            pattern.validate(value)
            return setattr(self, pattern.attribute, value)
        properties[key] = property(fget, fset)

    def __init__(self, **kwargs):
        # set all attributes found in the keyword arguments
        for key, value in kwargs.iteritems():
            if key in self.__attributes__:
                setattr(self, key, value)
        # set all attributes _NOT_ found to None
        for key, declaration in attributes.iteritems():
            if not hasattr(self, declaration.attribute):
                setattr(self, key, None)

    dct = dict(dct)
    dct.update(properties)
    dct['__init__'] = __init__
    dct['__attributes__'] = attributes
    return type(name, bases, dct)


# declarative class
class Artist(object):
    __metaclass__ = createClassFromDeclaration

    # FIXME: adding a second attribute changes the first pattern
    locale = Pattern('[A-Z]{2}-[A-Z]{2}')
    date = Pattern('[0-9]{4}-[0-9]{2}-[0-9]{2}')


# some unit tests
class TestArtist(unittest.TestCase):
    def test_attributes_are_default_initialized(self):
        artist = Artist()
        self.assertIsNone(artist.date)
        self.assertIsNone(artist.locale)

    def test_attributes_are_initialized_from_keywords(self):
        artist = Artist(locale="EN-US", date="2013-02-04")
        self.assertEqual(artist.date, "2013-02-04")
        # FIXME: the following does not work.
        # it validates against the date pattern
        self.assertEqual(artist.locale, "EN-US")

    def test_locale_with_valid_value(self):
        artist = Artist()
        artist.date = "2013-02-04"
        self.assertEqual(artist.locale, "2013-02-04")
        # FIXME: the following does not work.
        # it validates against the date pattern
        artist.locale = "EN-US"
        self.assertEqual(artist.locale, "EN-US")

    def test_locale_with_invalid_value_throws(self):
        artist = Artist()
        with self.assertRaises(ValueError):
            artist.locale = ""
        with self.assertRaises(ValueError):
            artist.locale = "EN-USA"


if __name__ == '__main__':
    unittest.main()

# vim: set ft=python sw=4 et sta:

When I comment out the second attribute (‘date’) the tests succeed, but with the second attribute the tests that try to set the first attribute (‘locale’) fail. What causes the unittests to fail?

Disclaimer: This code is only for training. There are ways to create the same functionality that do not involve metaclasses, properties and closures (as you and I know). But we don’t learn anything new if we only walk the streets we know. Please help me expand my Python knowledge.

Answer 1

The problem doesn’t really have anything to do with metaclasses or properties per se. It has to do with how you’re defining your get/set functions. Your fget and fset reference the variable pattern from the enclosing function. This creates a closure. The value of pattern will be looked up at the time fget/fset are called, not at the time they’re defined. So when you overwrite pattern on the next loop iteration, you cause all fget/fset functions to now reference the new pattern.

Here’s a simpler example that shows what’s going on:

def doIt(x):
    funs = []
    for key, val in x.iteritems():
        thingy = val + 1
        def func():
            return thingy
        funs.append(func)
    return funs

>>> dct = {'a': 1, 'b': 2, 'c': 3}
>>> funs = doIt(dct)
>>> for f in funs:
...     print f()

3
3
3

Notice that, even though the three functions are defined at times when thingy has different values, when I call them later they all return the same value. This is because they are all looking up thingy when they’re called, which is after the loop is done, so thingy just equals the last value it was set to.

The usual way to get around this is to pass in the variable you want to close over as the default value of an additional function argument. Try doing your getter and setter like this:

def fget(self, pattern=pattern):
    return getattr(self, pattern.attribute)
def fset(self, value, pattern=pattern):
    pattern.validate(value)
    return setattr(self, pattern.attribute, value)

Default arguments are evaluated at function definition time, not call time, so this forces each function to “save” the value of pattern it wants to use.