Question

I have basically the following setup in my package:

thing.py:

from otherthing import *

class Thing(Base):
    def action(self):
        ...do something with Otherthing()...

subthing.py:

from thing import *

class Subthing(Thing):
    pass

otherthing.py:

from subthing import *

class Otherthing(Base):
    def action(self):
        ... do something with Subthing()...

If I put all objects into one file, it will work, but that file would just become way too big and it’ll be harder to maintain. How do I solve this problem?

Answer 1

This is treading into the dreaded Python circular imports argument but, IMHO, you can have an excellent design and still need circular references.

So, try this approach:

thing.py:

class Thing(Base):
    def action(self):
        ...do something with otherthing.Otherthing()...

import otherthing

subthing.py:

import thing

class Subthing(thing.Thing):
    pass

otherthing.py:

class Otherthing(Base):
    def action(self):
        ... do something with subthing.Subthing()...

import subthing

There are a couple of things going on here. First, some background.

Due to the way importing works in Python, a module that is in the process of being imported (but has not been fully parsed yet) will be considered already imported when future import statements in other modules referencing that module are evaluated. So, you can end up with a reference to a symbol on a module that is still in the middle of being parsed – and if the parsing hasn’t made it down to the symbol you need yet, it will not be found and will throw an exception.

One way to deal with this is to use “tail imports”. The purpose of this technique is to define any symbols that other modules referring to this one might need before potentially triggering the import of those other modules.

Another way to deal with circular references is to move from from based imports to a normal import. How does this help? When you have a from style import, the target module will be imported and then the symbol referenced in the from statement will be looked up on the module object right at that moment.

With a normal import statement, the lookup of the reference is delayed until something does an actual attribute reference on the module. This can usually be pushed down into a function or method which should not normally be executed until all of your importing is complete.

The case where these two techniques don’t work is when you have circular references in your class hierarchy. The import has to come before the subclass definition and the attribute representing the super class must be there when the class statement is hit. The best you can do is use a normal import, reference the super class via the module and hope you can rearrange enough of the rest of your code to make it work.

If you are still stuck at that point, another technique that can help is to use accessor functions to mediate the access between one module and another. For instance, if you have class A in one module and want to reference it from another module but can’t due to a circular reference, you can sometimes create a third module with a function in it that just returns a reference to class A. If you generalize this into a suite of accessor functions, this doesn’t end up as much of a hack as it sounds.

If all else fails, you can move import statements into your functions and methods – but I usually leave that as the very last resort.

— EDIT —

Just wanted to add something new I discovered recently. In a “class” statement, the super class is actually a Python expression. So, you can do something like this:

>>> b=lambda :object
>>> class A(b()):
...     pass
... 
>>> a=A()
>>> a
<__main__.A object at 0x1fbdad0>
>>> a.__class__.__mro__
(<class '__main__.A'>, <type 'object'>)
>>> 

This allows you to define and import an accessor function to get access to a class from another class definition.