Question

I am testing an assembler I am writing which generates X86 instructions. I would like to do something like this to test whether the instructions work or not.

#include<stdio.h>

unsigned char code[2] = {0xc9, 0xc3};

int main() {
  void (*foo)();
  foo = &code;
  foo();
  return 0;
}

However it seems that OS X is preventing this due to DEP. Is there a way to either (a) disable DEP for this program or (b) enter the bytes in another format such that I can jump to them.

Answer 1

If you just need to test, try this instead, it’s magic…

const unsigned char code[2] = {0xc9, 0xc3};
^^^^^

The const keyword causes the compiler to place it in the const section (warning! this is an implementation detail!), which is in the same segment as the text section. The entire segment should be executable. It is probably more portable to do it this way:

__attribute__((section("text"))
const unsigned char code[2] = {0xc9, 0xc3};

And you can always do it in an assembly file,

    .text
    .globl code
code:
    .byte 0xc9
    .byte 0xc3

However: If you want to change the code at runtime, you need to use mprotect. By default, there are no mappings in memory with both write and execute permissions.

Here is an example:

#include <stdlib.h>
#include <sys/mman.h>
#include <err.h>
#include <stdint.h>

int main(int argc, char *argv[])
{
    unsigned char *p = malloc(4);
    int r;
    // This is x86_64 code
    p[0] = 0x8d;
    p[1] = 0x47;
    p[2] = 0x01;
    p[3] = 0xc3;
    // This is hackish, and in production you should do better.
    // Casting 4095 to uintptr_t is actually necessary on 64-bit.
    r = mprotect((void *) ((uintptr_t) p & ~(uintptr_t) 4095), 4096,
                 PROT_READ | PROT_WRITE | PROT_EXEC);
    if (r)
        err(1, "mprotect");
    // f(x) = x + 1
    int (*f)(int) = (int (*)(int)) p;
    return f(1);
}

The mprotect specification states that its behavior is undefined if the memory was not originally mapped with mmap, but you’re testing, not shipping, so just know that it works just fine on OS X because the OS X malloc uses mmap behind the scenes (exclusively, I think).