Question

In a python application I’m working on I would like to dynamically load packages (plugins) based on information provided at runtime (e.g. from a config file).

So, I change sys.path to add paths to the plugins I want to load, this generally works fine, but not if the plugin is a namespaced package inside a namespace which has already been initialized (I assume that is the problem anyway).

Example:

# lib1 contains plugins/__init__.py and plugins/foo/__init__.py
# lib2 contains plugins/__init__.py and plugins/bar/__init__.py
# plugins is a namespace package

import sys

sys.path.append ('lib1')
import plugins.foo

sys.path.append ('lib2')
import plugins.bar

The above code fails with an ImportError, presumably because the “import plugins.foo” line initialized the plugins namespace/package and no further attempt is made to search sys.path for other packages in the namespace.

If I change the code to this:

sys.path.append ('lib1')
sys.path.append ('lib2')
import plugins.foo
import plugins.bar

Both imports work, but I’d like to add to sys.path after the plugins.foo import.

So, my questions are:

1. Is my assumption correct that the second import fails because of the
   namespace package?
2. Is there a work-around?

Answer 1

I think your diagnosis is right. When you import plugins.foo, python also loads plugins and caches it in sys.modules. I haven’t replicated your set-up, but I’d try del sys.modules['plugins'] before you import bar. If that doesn’t work, try reloading plugins:

sys.path.append ('lib2')
import plugins
reload(plugins)
import plugins.bar

I must admit that reloading is, by general agreement, not a good idea in running programs. The “right” solution would be to figure out a workflow that lets you set the full sys.path before you start importing from the package. Or perhaps not to spread one package over different places.