Question

(I’m writing in pure x86 assembly for NASM, not in C/C++.)

I’m getting a segmentation fault when the compiled binary runs – I’m aware this is an age old error message, but searching for this specific instance of a seg fault hasn’t proved to be fruitful:

gdb suggests that the fault happened at ENTER 616,0 at the beginning of a call. I believe it’s the same as pushing the old %rbp, storing %rsp into %rbp, and decreasing %rsp for 616 bytes of local variables.

Does anyone with more experience have hints as to why a segmentation fault can happen here? It seems like a strange place for memory access issues – the only thing that comes to mind is that 616 might be a lot to decrease the value by, but other than that it’s baffling me. Is there a limit on the size allowed (other than the total amount of memory available)?

Any help would be much appreciated.

Update:
If it helps, this isn’t the end of a long series of recursive calls:

(gdb) backtrace
#0  0x00000000004005e0 in user_func ()
#1  0x0000000000400e69 in if4 ()
#2  0xffffffffffffffff in ?? ()
#3  0xffffffffffffffff in ?? ()
#4  0x0000000000000000 in ?? ()

(gdb) frame 0
#0  0x00000000004005e0 in user_func ()

(gdb) disassemble
Dump of assembler code for function user_partition:
=> 0x00000000004005e0 <+0>:     enterq $0x2b0,$0x0
   0x00000000004005e4 <+4>:     push   %r15
   0x00000000004005e6 <+6>:     push   %r12
   0x00000000004005e8 <+8>:     push   %r13
   0x00000000004005ea <+10>:    push   %r14
   ...

Update 2:
Since the backtrace seems to indicate a corrupted stack pointer, here’s some relevant details on what each method call looks like:

user_func:                           
   ENTER      296, 0                         
   PUSH       R13      ; Saving any callee-saved registers used in main body   
   PUSH       R15     
   PUSH       R14     
   PUSH       R12     
   ; Only opcodes to MOV between temp registers and [ RBP - x ]
   MOV        RAX, 0                        
   POP        R12      ; Restoring the callee-saved registers    
   POP        R14
   POP        R15
   POP        R13
   LEAVE                                    
   RET                                      

Is it possible that I’ve done something wrong here in terms of pushing / popping things? The POP before LEAVE seemed right to me…

Answer 1

Values like 0x0000000000000 and 0xffffffffffff in your backtrace indicate that you’ve trashed the stack at some point (overwritten a return value or similar). It is likely that your stack pointer is garbage, hence a high likelihood that pushing to it will cause a seg-fault.