Question

I’m looking for techniques that allow users to override modules in an application or extend an application with new modules.

Imagine an application called pydraw. It currently provides a Circle class, which inherits Shape. The package tree might look like:

/usr/lib/python/
└── pydraw
    ├── __init__.py
    ├── shape.py
    └── shapes
        ├── circle.py
        └── __init__.py

Now suppose I’d like to enable dynamic discovery and loading of user modules that implement a new shape, or perhaps even the Shape class itself. It seems most straightforward for a user’s tree to have the same structure as the application tree, such as:

/home/someuser/python/
└── pydraw
    ├── __init__.py
    ├── shape.py       <-- new superclass
    └── shapes
        ├── __init__.py
        └── square.py   <-- new user class

In other words, I’d like to overlay and mask an application tree with same-named files from the user’s tree, or at least get that apparent structure from a Python point of view.

Then, by configuring sys.path or PYTHONPATH, pydraw.shapes.square might be discoverable. However, Python’s module path search doesn’t find modules such as square.py. I presume this is because __method__ already contains a parent module at another path.

How would you accomplish this task with Python?

Answer 1

If you want to load python code dynamically from different locations, you can extend the search __path__ attributes by using the pkgutil module:

By placing these lines into each pydraw/__init__.py and pydraw/shapes/__init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

You will be able to write import statement as if you had a unique package:

>>> import pydraw.shapes
>>> pydraw.shapes.__path__
['/usr/lib/python/pydraw/shapes', '/home/someuser/python/pydraw/shapes']
>>> from pydraw.shapes import circle, square
>>>

You may think about auto-registration of your plugins. You can still use basic python code for that by setting a module variable (which will act as a kind of singleton pattern).

Add the last line in every pydraw/shapes/__init__.py file:

 from pkgutil import extend_path
 __path__ = extend_path(__path__, __name__)

 # your shape registry
 __shapes__ = []

You can now register a shape in top of its related module (circle.py or square.py here).

 from pydraw.shapes import __shapes__
 __shapes__.append(__name__)

Last check:

 >>> from pydraw.shapes import circle,square
 >>> from pydraw.shapes import circle,square,__shapes__
 >>> __shapes__
 ['pydraw.shapes.circle', 'pydraw.shapes.square']