Question

I am relatively new to python (already did some 1h scripts like a little webserver or a local network chat) and want to program a plugin manager in it.
My idea is, that there is an interface for plugins, that has the following features:

getDependencies -> all dependencies of the plugin to other plugins
getFunctions -> all functions that this plugin introduces
initialize -> a function that is called when loading the plugin

(I could imagine to have a topological sorting algorithm on the dependencies to decide the order in which the plugins are initialized.)

I would like to implement multithreading, meaning that each plugin runs in its own thread, that has a working queue of function-calls that will be executed serially. When a plugin calls the function of another plugin it calls the manager who will in turn insert the function-call into the queue of the other plugin.

Further the manager should provide some kind of event system in which the plugins can register their own events and become listeners to the events of others.

Also I want to be able to reload a plugin if the code has changed or its thread crashed, without shutting down the manager/application. I already read How do I unload (reload) a Python module? in conjunction with this.

To make it clear once more: The manager should not provide any other functionality than supporting its plugins with a common communication interface to each other, the ability to run side by side (in a multithreaded manner without requiring the plugins to be aware of this) and restoring updated/crashed plugins.

So my questions are: Is it possible to do this in python? And if yes are there design mistakes in this rough sketch? I would appreciate any good advice on this.

Other “literature”:
Implementing a Plugin System in Python

Answer 1

At the most basic level, first of all, you want to provide a basic Plugin class which is a base for all plugins written for your application.

Next we need to import them all.

class PluginLoader():
    def __init__(self, path):
        self.path = path

    def __iter__(self):
        for (dirpath, dirs, files) in os.walk(self.path):
            if not dirpath in sys.path:
                sys.path.insert(0, dirpath)
        for file in files:
                (name, ext) = os.path.splitext(file)
                if ext == os.extsep + "py":
                    __import__(name, None, None, [''])
        for plugin in Plugin.__subclasses__():
            yield plugin

In Python 2.7 or 3.1+, instead of __import__(name, None, None, ['']), consider:

import importlib  # just once
importlib.import_module(name)

This loads every plugin file and gives us all plugins. You would then select your plugins as you saw fit, and then use them:

from multiprocessing import Process, Pipe

plugins = {}

for plugin in PluginLoader("plugins"):
    ... #select plugin(s)
    if selected:
        plugins[plugin.__name__], child = Pipe()
        p = Process(target=plugin, args=(child,))
        p.start()

...

for plugin in plugins.values():
    plugin.put("EventHappened")

...

for plugin in plugins.values():
    event = plugin.get(False)
    if event:
       ... #handle event

This is just what comes to mind at first. Obviously much more would be needed for flesh this out, but it should be a good basis to work from.