Question

I’m trying to debug a memory leak (see question Memory leak in Python Twisted: where is it?).

When the garbage collector is running, does it have access to all Python objects created by the Python interpreter? If we suppose Python C libraries are not leaking, should RSS memory usage grow linearly with respect to the GC object count? What about sys.getobjects?

Answer 1

CPython uses two mechanisms to clean up garbage. One is reference counting, which affects all objects but which can’t clean up objects that (directly or indirectly) refer to each other. That’s where the actual garbage collector comes in: python has the gc module, which searches for cyclic references in objects it knows about. Only objects that can potentially be part of a reference cycle need to worry about participating in the cyclic gc. So, for example, lists do, but strings do not; strings don’t reference any other objects. (In fact, the story is a little more complicated, as there’s two ways of participating in cyclic gc, but that isn’t really relevant here.)

All Python classes (and instances thereof) automatically get tracked by the cyclic gc. Types defined in C aren’t, unless they put in a little effort. All the builtin types that could be part of a cycle do. But this does mean the gc module only knows about the types that play along.

Apart from the collection mechanism there’s also the fact that Python has its own aggregating memory allocator (obmalloc), which allocates entire memory arenas and uses the memory for most of the smaller objects it creates. Python now does free these arenas when they’re completely empty (for a long time it didn’t), but actually emptying an arena is fairly rare: because CPython objects aren’t movable, you can’t just move some stragglers to another arena.