I was just reading through a Phrack article which had disassembly in HP-UX. I read that there are two classes of functions which are possible under HP-UX and SPARC; leaf and non-leaf functions. Following is a section of the disassembly I have taken from here.
(gdb) disass leaf
Dump of assembler code for function foo:
0x3280 <leaf>: copy r3,r1
0x3284 <leaf+4>: copy sp,r3
0x3288 <leaf+8>: stw,ma r1,40(sr0,sp)
0x328c <leaf+12>: stw r26,-24(sr0,r3)
0x3290 <leaf+16>: stw r0,8(sr0,r3)
0x3294 <leaf+20>: ldi 1,r19
0x3298 <leaf+24>: stw r19,8(sr0,r3)
0x329c <leaf+28>: ldo 40(r3),sp
0x32a0 <leaf+32>: ldw,mb -40(sr0,sp),r3
0x32a4 <leaf+36>: bv,n r0(rp)
End of assembler dump.
(gdb)
Usually when a function is called, the return address is pushed onto the stack so that the program knows where to return the control to, once the function has finished executing. How does that work in the case of these leaf functions?
I dont have any access of HP-UX/SPARC machines so have no way to try this out by myself(and I dont understand the assembly in this case very well either, because of the same reason).
Could someone explain how control returns to the callee function in this case?
First, the document you link to, and the code you show, are not Sparc but PA-RISC, which is a distinct architecture. As far as I know, there is no version of HP/UX which runs on Sparc-based systems.
Nevertheless, the point about leaf functions is similar across many architectures, including Sparc, PA-RISC, PowerPC, ARM, MIPS… in fact all RISC architectures. For all of those, the opcode which performs a function call does not store the return address on the stack; in fact, there is no “real” stack as known by the hardware. Instead, there is just a software convention for using a specific register as stack pointer. The calling opcode stores the returned address in a specific register, usually called the “link register”. The opcode for returning from a function just reads that register.
If the function itself (let’s call it A) calls another function (B), then that nested call will also use the link register; however, A will need the contents of the link register when it will return. Therefore, A must save the link register somewhere, usually in memory, and more precisely in the memory area conventionally used as “the stack”.
A leaf function is a function which calls no other function; it just does its job and returns. A leaf function needs not save the link register on the stack since nothing will alter the contents of the link register.
An additional constraint, which is also conventional, is about stack frame structure. Some operating systems, on some architectures, insist on functions using the stack in a very specific manner (called “stack frames”), which can be reliably explored by debuggers. The specification then calls for saving the return address in a precise slot on the stack, and the called function must do that very early in its execution. Then, the same operating systems may state that this is not absolutely required for leaf functions (it is easier for the debugger to deal with a frame-less function if that function calls none other, i.e. there is no frame pushed on the stack over the one which is missing). Not setting up a stack frame is more efficient (in execution speed and code size) than setting up a stack frame, hence the exception.