Question

In my application, I have:

...
folder_a/
    __init__.py
    a.py
    folder_b/
        __init__.py
        b1.py 
        b2.py
        ...

and I need to dynamically load modules from folder_b into a.py, based on an variable, say, module_from_b. To do this, in a.py, I do:

mymodule = __import__("folder_b.%s" % module_from_b, globals(), locals(), fromlist=["*"])

Since this is going to live inside a loop, i.e. be called numerous times, I wonder if it has any impact on the performance, or anything else? If so, is there any way to mitigate it while preserving the capability of dynamic module loading?

Answer 1

I’ve never benchmarked it, but I believe importing (whether dynamic or static) a module that has been previously imported into the running interpreter (even by completely unrelated code) should be pretty cheap. All it really has to do should be some (successful) dictionary lookups.

Importing a new module will of course run all the code in the module, plus the file system search to find it.

So if you’re repeatedly importing a dynamically chosen module from a relatively small set of modules, then you shouldn’t have much of a problem so long as you can tolerate a delay the first time each specific module is used (how much of a delay depends on your modules); after a little while almost all of the modules you’re importing will already have been imported, and so the __import__ calls will become cheap.

An alternative design you might consider: If the total set of modules you’ll need is knowable up front (either statically or dynamically), you could pre-import them before your loop to “warm up” Python’s imported module set. If there aren’t many modules in b that won’t be used you could import everything in b’s __init__.py.

That way the import delays would be gotten out of the way at initiation time and you could use getattr on the b package to dynamically get your modules instead of needing to use __import__. Not such a good option if you’re loading so many modules that you want to spread out the import costs over your loop, or if you have many modules but only need a relatively small number of them and it’s hard to know which ones ahead of time.

There’s another possibly even better approach, which will work if in the places where you’re identifying which module to import you’re choosing some constant string (and not, say, something read from a configuration file or user input). Rather than passing the name of the module onwards to eventually be imported elsewhere, why not import the module then and there and pass the module itself on to eventually be used elsewhere? As a made up example, since I don’t know what you’re doing, instead of:

for module_name in ['b1', 'b2', 'b3', 'b4']:
    function_using_module(module_name)

def function_using_module(module_name):
    module = __import__(...)
    ...

You could do:

from folder_b import b1, b2, b3, b4

for module in [b1, b2, b3, b4]:
    function_using_module(module)

def function_using_module(module):
    ...

Modules are objects just like anything else, so you can store them in lists or objects or whatever you’re doing with the names. Passing the modules around directly is usually cleaner than passing the names around as proxies for the modules and later importing them.