Question

Somehow the memory my Python program takes more and more memory as it runs (the VIRT and RES) column of the “top” command keep increasing.

However, I double checked my code extremely carefully, and I am sure that there is no memory leaks (didn’t use any dictionary, no global variables. It’s just a main method calling a sub method for a number of times).

I used heapy to profile my memory usage by

from guppy import hpy;
heap = hpy();
.....
print heap.heap();

each time the main method calls the sub method. Surprisingly, it always gives the same output. But the memory usage just keeps growing.

I wonder if I didn’t use heapy right, or VIRT and RES in “top” command do not really reflect the memory my code uses?

Or can anyone provide a better way to track down the memory usage in a Python script?

Thanks a lot!

Answer 1

Two possible cases:

• your function is pure Python, in which case possible causes include

  • you are storing an increasing number of large objects
  • you are having cycles of objects with a __del__ method, which the gc won’t touch

  I’d suggest using the gc module and the gc.garbage and gc.get_objects function (see http://docs.python.org/library/gc.html#module-gc), to get list of existing objects, and you can then introspect them by looking at the __class__attribute of each object for instance to get information about the object’s class.

• your function is at least partially written in C / C++, in which case the problem potentially is in that code. The advice above still applies, but won’t be able to see all leaks: you will see leaks caused by missing calls to PY_DECREF, but not low level C/C++ allocations without a corresponding deallocation. For this you will need valgrind. See this question for more info on that topic