Question

In order to find more easily buffer overflows I am changing our custom memory allocator so that it allocates a full 4KB page instead of only the wanted number of bytes. Then I change the page protection and size so that if the caller writes before or after its allocated piece of memory, the application immediately crashes.

Problem is that although I have enough memory, the application never starts up completely because it runs out of memory. This has two causes:

• since every allocation needs 4 KB, we probably reach the 2 GB limit very soon. This problem could be solved if I would make a 64-bit executable (didn’t try it yet).
• even when I only need a few hundreds of megabytes, the allocations fail at a certain moment.

The second problem is the biggest one, and I think it’s related to the maximum number of PTE’s (page table entries, which store information on how Virtual Memory is mapped to physical memory, and whether pages should be read-only or not) you can have in a process.

My questions (or a cry-for-tips):

• Where can I find information about the maximum number of PTE’s in a process?
• Is this different (higher) for 64-bit systems/applications or not?
• Can the number of PTE’s be configured in the application or in Windows?

Thanks,

Patrick

PS. note for those who will try to argument that you shouldn’t write your own memory manager:

• My application is rather specific so I really want full control over memory management (can’t give any more details)
• Last week we had a memory overwrite which we couldn’t find using the standard C++ allocator and the debugging functionality of the C/C++ run time (it only said “block corrupt” minutes after the actual corruption”)
• We also tried standard Windows utilities (like GFLAGS, …) but they slowed down the application by a factor of 100, and couldn’t find the exact position of the overwrite either
• We also tried the “Full Page Heap” functionality of Application Verifier, but then the application doesn’t start up either (probably also running out of PTE’s)

Answer 1

A shotgun approach is to allocate those isolated 4KB entries at random. This means that you will need to rerun the same tests, with the same input repeatedly. Sometimes it will catch the error, if you’re lucky.

A slightly smarter approach is to use another algorithm than just random – e.g. make it dependent on the call stack whether an allocation is isolated. Do you trust std::string users, for instance, and suspect raw malloc use?