Question

Having read this interesting article outlining a technique for debugging heap corruption, I started wondering how I could tweak it for my own needs. The basic idea is to provide a custom malloc() for allocating whole pages of memory, then enabling some memory protection bits for those pages, so that the program crashes when they get written to, and the offending write instruction can be caught in the act. The sample code is C under Linux (mprotect() is used to enable the protection), and I’m curious as to how to apply this to native C++ and Windows. VirtualAlloc() and/or VirtualProtect() look promising, but I’m not sure how a use scenario would look like.

Fred *p = new Fred[100];
ProtectBuffer(p);
p[10] = Fred(); // like this to crash please

I am aware of the existence of specialized tools for debugging memory corruption in Windows, but I’m still curious if it would be possible to do it “manually” using this approach.

EDIT: Also, is this even a good idea under Windows, or just an entertaining intellectual excercise?

Answer 1

Yes, you can use VirtualAlloc and VirtualProtect to set up sections of memory that are protected from read/write operations.

You would have to re-implement operator new and operator delete (and their [] relatives), such that your memory allocations are controlled by your code.

And bear in mind that it would only be on a per-page basis, and you would be using (at least) three pages worth of virtual memory per allocation – not a huge problem on a 64-bit system, but may cause problems if you have many allocations in a 32-bit system.

Roughly what you need to do (you should actually find the page-size for the build of Windows – I’m too lazy, so I’ll use 4096 and 4095 to represent pagesize and pagesize-1 – you also will need to do more error checking than this code does!!!):

void *operator new(size_t size)
{
    Round size up to size in pages + 2 pages extra.
    size_t bigsize = (size + 2*4096 + 4095) & ~4095; 

    // Make a reservation of "size" bytes. 
    void *addr = VirtualAlloc(NULL, bigsize, PAGE_NOACCESS, MEM_RESERVE);

    addr = reinterpret_cast<void *>(reinterpret_cast<char *>(addr) + 4096);

    void *new_addr = VirtualAlloc(addr, size, PAGE_READWRITE, MEM_COMMIT); 

    return new_addr;
}

void operator delete(void *ptr)
{
    char *tmp = reinterpret_cast<char *>(ptr) - 4096;

    VirtualFree(reinterpret_cast<void*>(tmp)); 
}

Something along those lines, as I said – I haven’t tried compiling this code, as I only have a Windows VM, and I can’t be bothered to download a compiler and see if it actually compiles. [I know the principle works, as we did something similar where I worked a few years back].