Question

I’m working on trying to get a ruby extension written in c, working with ruby 1.9.2 on Mac OSX. I keep getting a segfault caused by a 64-bit pointer returned from a function being truncated to a 32-bit value and trying to access low memory.

Here’s what I’ve figured out about it. VALUE is a long, defined in the ruby source code, and the code won’t compile unless sizeof(long) is 8 bytes.

In file_b.c:

VALUE
rb_helper_function_foo(VALUE file)
{
  VALUE blah;

  ...

  //Using gdb, blah right here is a 64bit address, ex: 0x1 009d 62a0
  return blah;
}

In file_a.c

VALUE
func_do_stuff(int argc, VALUE *argv, VALUE obj)
{
  VALUE filename, file_path;

  ...

  // But here, the size of what is returned is 4 bytes
  // file_path has a size of 8 bytes though
  file_path = rb_helper_function_foo(filename);

  //This statement will segfault :-(
  if(!RTEST(file_path))
  {
    ...
  }
  ...
}

At the end of the function call, the return value has the correct address, but only the lower 4 bytes get returned: 0x009d52a0 instead of 0x1009d52a0. Accessing the low memory address is causing a segfault.

In the assembly code that gets called during the return, there’s this instruction

movslq %eax, %r12

which only copies the lower 4 bytes, but I need all 8 bytes of %rax to be copied.

I’m on ruby 1.9.2-p290 (though it did the same with p180), Mac OSX 10.6.8, using xcode 4.0.1 (with gcc 4.2.1), but I also tried upgrading to gcc 4.6.1. It has also previously tried xcode 3.2.

Thanks for any help you can give me.

Answer 1

Maybe this is as simple as rb_helper_function_foo not being declared in the file_a.c compilation unit– so the compiler is assuming it returns int instead of VALUE?