So I was asking myself what would happen if I tried to do a heap overflow on Windows XP, and I was surprise to see that, once the program “ate” all the RAM (this happens instantly, by the way), the size of the process in the task manager goes down to 5MB and doesn’t move afterwards. The computer memory usage is still growing, however.
So why is Windows not able to see that my software takes GB of memory ? I feel like it can be a security problem because once a software ate all the memory, it can “hide” in the small process groups (and maybe I’m a little bit paranoid).
Note : nothing happens when the heap is full, the cpu just jumps to 100% because my for(;;) loop runs like crazy once malloc fails.
Edit : Ok! Never knew that you could tweak the task manager columns. I learnt something today :D.
Interesting experiment .. by default the Task Manager shows default working set. There are other memory fields, such as Paged and Unpaged pools and Working sets. Page faults can also tell you that the program is trying allocate memory but failing.