Question

EAX is used to store return value of function in 32bit platform, I just wonder if the size of return value of a function is bigger than 4 bytes, how does eax handle it? In this case OS can save the return value on the stack and store the address of the stack in EAX, but then how can OS tell whether the value stored in EAX is an address to the return value or is actually the return value itself?

Answer 1

The caller and callee have to agree on what the registers and stack contain. This is called the calling convention, which is part of a larger concept called the application binary interface (ABI). The callee defines how it wants to be called (ie. whether arguments need to be on the stack, in registers, etc.) and the compiler ensures that the code it generates complies with the calling convention.

As for your specific question, it depends the ABI. Sometimes if the return value is larger than 4 bytes but not larger than 8 bytes, it can be split into EAX and EDX. But most of the time the calling function will just allocate some memory (usually on the stack) and pass a pointer to this area to the called function.

Note also that the role of the OS is not as important as you appear to think. Binaries with different calling conventions may coexist on the same system, and binaries can even use different calling conventions internally. The ABI of the OS is only important when the binary calls its system libraries.