Question

I need a way to find the dependencies for each of my Python package’s sub-modules at runtime so I can initialize them in a proper order (see my current [EDIT: former] solution here, which doesn’t work to well), so at first I used the standard Python module modulefinder, but that was way too slow (~1-2 seconds per module).

My next choice was to analyze all the globals of each module, and find from those globals which sub-module each sub-module depends upon. (This is my current solution EDIT: I have a better solution now – see my answer). This algorithm is much faster than modulefinder (it takes <200ms per module), but it only works for relative imports, instead of the fully-qualified import style, which is unacceptable.

So, what I need is either:

• A quicker alternative to modulefinder
• An alternative algorithm

NOTE: I call my dependency analyzer at the start of each module, like so:

# File my_package/module3.py

import my_package.module1 # Some misc. module
import my_package.module2 # Some other misc. module
import my_package.dependency_analyzer

my_package.dependency_analyzer.gendeps()

(Just in case it helps you any.)

Thank you!

EDIT: I have a solution now – see my answer.

Answer 1

I think I have a solution to my own question 🙂

Here’s what would go into the dependency_analyzer module talked about above:

import sys
from sys import _getframe as getframe
import atexit

examined_modules = []

def gendeps():
    """Adds the calling module to the initialization queue."""
    # Get the calling module's name, and add it to the intialization queue
    calling_module_name = getframe(1).f_globals['__name__']
    examined_modules.append(calling_module_name)

def init():
    """Initializes all examined modules in the correct order."""

    for module in examined_modules:
        module = sys.modules[module]
        if hasattr(module, 'init'):
            module.init()
        if hasattr(module, 'deinit'):
            # So modules get de-initialized in the correct order,
            # as well
            atexit.register(module.deinit)

Now, at the start of each module (after all the import statements – this is crucial), a call to gendeps is placed. This algorithm works because each time a module is imported, that call to gendeps is executed. However, since all of the import statements are placed before the call to gendeps in your own module, the least-dependent modules are placed in the initialization queue first, and the most-dependent modules are placed in the initialization queue last.