Question

tl;dr: How come property decorators work with class-level function definitions, but not with module-level definitions?

I was applying property decorators to some module-level functions, thinking they would allow me to invoke the methods by mere attribute lookup.

This was particularly tempting because I was defining a set of configuration functions, like get_port, get_hostname, etc., all of which could have been replaced with their simpler, more terse property counterparts: port, hostname, etc.

Thus, config.get_port() would just be the much nicer config.port

I was surprised when I found the following traceback, proving that this was not a viable option:

TypeError: int() argument must be a string or a number, not 'property'

I knew I had seen some precedant for property-like functionality at module-level, as I had used it for scripting shell commands using the elegant but hacky pbs library.

The interesting hack below can be found in the pbs library source code. It enables the ability to do property-like attribute lookups at module-level, but it’s horribly, horribly hackish.

# this is a thin wrapper around THIS module (we patch sys.modules[__name__]).
# this is in the case that the user does a "from pbs import whatever"
# in other words, they only want to import certain programs, not the whole
# system PATH worth of commands.  in this case, we just proxy the
# import lookup to our Environment class
class SelfWrapper(ModuleType):
    def __init__(self, self_module):
        # this is super ugly to have to copy attributes like this,
        # but it seems to be the only way to make reload() behave
        # nicely.  if i make these attributes dynamic lookups in
        # __getattr__, reload sometimes chokes in weird ways...
        for attr in ["__builtins__", "__doc__", "__name__", "__package__"]:
            setattr(self, attr, getattr(self_module, attr))

        self.self_module = self_module
        self.env = Environment(globals())

    def __getattr__(self, name):
        return self.env[name]

Below is the code for inserting this class into the import namespace. It actually patches sys.modules directly!

# we're being run as a stand-alone script, fire up a REPL
if __name__ == "__main__":
    globs = globals()
    f_globals = {}
    for k in ["__builtins__", "__doc__", "__name__", "__package__"]:
        f_globals[k] = globs[k]
    env = Environment(f_globals)
    run_repl(env)

# we're being imported from somewhere
else:
    self = sys.modules[__name__]
    sys.modules[__name__] = SelfWrapper(self)

Now that I’ve seen what lengths pbs has to go through, I’m left wondering why this facility of Python isn’t built into the language directly. The property decorator in particular seems like a natural place to add such functionality.

Is there any partiuclar reason or motivation for why this isn’t built directly in?

Answer 1

This is related to a combination of two factors: first, that properties are implemented using the descriptor protocol, and second that modules are always instances of a particular class rather than being instantiable classes.

This part of the descriptor protocol is implemented in object.__getattribute__ (the relevant code is PyObject_GenericGetAttr starting at line 1319). The lookup rules go like this:

1. Search through the class mro for a type dictionary that has name
2. If the first matching item is a data descriptor, call its __get__ and return its result
3. If name is in the instance dictionary, return its associated value
4. If there was a matching item from the class dictionaries and it was a non-data descriptor, call its __get__ and return the result
5. If there was a matching item from the class dictionaries, return it
6. raise AttributeError

The key to this is at number 3 – if name is found in the instance dictionary (as it will be with modules), then its value will just be returned – it won’t be tested for descriptorness, and its __get__ won’t be called. This leads to this situation (using Python 3):

>>> class F:
...    def __getattribute__(self, attr):
...      print('hi')
...      return object.__getattribute__(self, attr)
... 
>>> f = F()
>>> f.blah = property(lambda: 5)
>>> f.blah
hi
<property object at 0xbfa1b0>

You can see that .__getattribute__ is being invoked, but isn’t treating f.blah as a descriptor.

It is likely that the reason for the rules being structured this way is an explicit tradeoff between the usefulness of allowing descriptors on instances (and, therefore, in modules) and the extra code complexity that this would lead to.