TL/DR:
import gc, sys
print len(gc.get_objects()) # 4073 objects in memory
# Attempt to unload the module
import httplib
del sys.modules["httplib"]
httplib = None
gc.collect()
print len(gc.get_objects()) # 6745 objects in memory
UPDATE
I’ve contacted Python developers about this problem and indeed it’s not going to be possible to unload a module completely “in next five years”. (see the link)
Please accept that Python indeed does not support unloading modules for severe, fundamental, insurmountable, technical problems, in 2.x.
During my recent hunt for a memleak in my app, I’ve narrowed it down to modules, namely my inability to garbage collect an unloaded module. Using any method listed below to unload a module leaves thousands of objects in memory. In other words – I can’t unload a module in Python…
The rest of the question is attempt to garbage collect a module somehow.
Let’s try:
import gc
import sys
sm = sys.modules.copy() # httplib, which we'll try to unload isn't yet
# in sys.modules, so, this isn't the source of problem
print len(gc.get_objects()) # 4074 objects in memory
Let’s save a copy of sys.modules to attempt to restore it later.
So, this is a baseline 4074 objects. We should ideally return to this somehow.
Let’s import a module:
import httplib
print len(gc.get_objects()) # 7063 objects in memory
We’re up to 7K non-garbage objects.
Let’s try removing httplib from sys.modules.
sys.modules.pop('httplib')
gc.collect()
print len(gc.get_objects()) # 7063 objects in memory
Well, that didn’t work. Hmm, but isn’t there a reference in __main__? Oh, yeah:
del httplib
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory
Hooray, down 300 objects. Still, no cigar, that’s way more than 4000 original objects.
Let’s try restoring sys.modules from copy.
sys.modules = sm
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory
Hmmm, well that was pointless, no change..
Maybe if we wipe out globals…
globals().clear()
import gc # we need this since gc was in globals() too
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory
locals?
locals().clear()
import gc # we need this since gc was in globals() too
gc.collect()
print len(gc.get_objects()) # 6746 objects in memory
What the.. what if we imported a module inside of exec?
local_dict = {}
exec 'import httplib' in local_dict
del local_dict
gc.collect()
print len(gc.get_objects()) # back to 7063 objects in memory
Now, that’s not fair, it imported it into __main__, why? It should have never left the local_dict… Argh! We back to fully imported httplib.
Maybe if we replaced it with a dummy object?
from types import ModuleType
import sys
print len(gc.get_objects()) # 7064 objects in memory
Bloody…..!!
sys.modules['httplib'] = ModuleType('httplib')
print len(gc.get_objects()) # 7066 objects in memory
Die modules, die!!
import httplib
for attr in dir(httplib):
setattr(httplib, attr, None)
gc.collect()
print len(gc.get_objects()) # 6749 objects in memory
Okay, after all attempts, the best is +2675 (nearly +50%) from starting point… That’s just from one module… That doesn’t even have anything big inside…
Ok, now seriously, where’s my error?
How do I unload a module and wipe out all of it’s contents?
Or is Python’s modules one giant memory leak?
Full source in simpler to copy form: http://gist.github.com/450606
Python does not support unloading modules.
However, unless your program loads an unlimited number of modules over time, that’s not the source of your memory leak. Modules are normally loaded once at start up and that’s it. Your memory leak most likely lies elsewhere.
In the unlikely case that your program really does load an unlimited number of modules over time, you should probably redesign your program. 😉