Question

Short question:
Is it possible to map a buffer that has been malloc’d to have two ways (two pointers pointing to the same physical memory) of accessing the same buffer?
Or, is it possible to temporarily move a virtual memory address received by malloc? Or is it possible to point from one location in virtual space to another?

Background:
I am working with DirectFB, a surface management and 2D graphics composting library. I am trying to enforce the Locking protocol which is to Lock a surface, modify the memory only while locked (the pointer is to system memory allocated using malloc), and unlocking the surface.

I am currently trying to trace down a bug in an application that is locking a surface and then storing the pixel pointer and modifying the surface later. This means that the library does not know when it is safe to read or write to a surface. I am trying to find a way to detect that the locking protocol has been violated. What I would like is a way to invalidate the pointer passed to the user after the unlock call is made. Even better, I would like the application to seg fault if it tries to access the memory after the lock. This would stop in the debugger and give us an idea of which surface is involved, which routine is involved, who called it, etc.

Possible solutions:

1. Create a temporary buffer, pass the buffer pointer to the user, on unlock copy the pixels to the actual buffer, delete the temporary
   buffer.

   • Pros: This is an implementable solution.
   • Cons: Performance is slow as it requires a copy which is expensive, also the memory may or may not be available. There is no
     way to guarantee that one temporary surface overlaps another allowing
     an invalidated pointer to suddenly work again.

2. Make an additional map to a malloc’d surface and pass that to the user. On unlock, unmap the memory.

   • Pros: Very fast, no additional memory required.
   • Cons: Unknown if this is possible.
   • Gotchas: Need to set aside a reserved range of addresses are never used by anything else (including malloc or the kernel). Also need to
     ensure that no two surfaces overlap which could allow an old pointer
     to suddenly point to something valid and not seg fault when it should.

3. Take advantage of the fact that the library does not access the memory while locked by the user and simply move the virtual address on
   a lock and move it back on an unlock.

   • Pros: Very fast, no additional memory required.
   • Cons: Unknown if this is possible.
   • Gotchas: Same as “2” above.

Is this feasible?

Additional info:

• This is using Linux 2.6, using stdlib.
• The library is written in C.
• The library and application run in user space.
• There is a possibility of using a kernel module (to write a custom memory allocation routine), but the difficulty of writing a module in
  my current working climate would probably reduce the chances to near
  zero levels that I could actually implement this solution. But if this
  is the only way, it would be good to know.
• The underlying processor is x86.

Answer 1

The function you want to create multiple mappings of a page is shm_open.

You may only be using the memory within one process, but it’s still “shared memory” – that is to say, multiple virtual mappings for the same underlying physical page will exist.

However, that’s not what you want to do. What you should actually do is have your locking functions use the mprotect system call to render the memory unreadable on unlock and restore the permissions on lock; any access without the lock being held will cause a segfault. Of course, this’ll only work with a single simultaneous accessing thread…

Another, possibly better, way to track down the problem would be to run your application in valgrind or another memory analysis tool. This will greatly slow it down, but allows you very fine control: you can have a valgrind script that will mark/unmark memory as accessible and the tool will kick you straight into the debugger when a violation occurs. But for one-off problem solving like this, I’d say install an #ifdef DEBUG-wrapped mprotect call in your lock/unlock functions.