Question

Can anyone explain to me what this means?

‘Run-Time Check Failure #0 – The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.’

Answer 1

When calling a function, the compiler has to push some arguments on the stack, or put them in some registers. The function body will change some memory location (or a register) to contain the return value. Then it will return to a block of code at a location stored ‘somewhere’ on the stack.

The calling convention specifies the concrete details: e.g. first push the return address, then push arguments (input or output) on the stack from left to right, then execute the function, pop the arguments off again, then pop the return address and jump to that location.

If the caller does this differently than the function expects, things can go wrong (return location is not at the expected stack position).

ESP is typically the register containing the address of the current stack frame. This register is used in combination with indexes to obtain arguments in the function body. When returning, the stack top is typically reset to ESP, and the processor jumps to the location at e.g. ESP+1.

Things that may cause this to happen:

• someone wrote over the values of the stack and changed the return location (e.g. buffer overflow)
• the caller had a different calling convention than the callee