Question

On x86 after a call the address that is pushed to the stack points to the place after the call instruction. This instruction can be of a variable length on x86 machines. Is there a way to detect which kind of the call was used?

For example, indirect call *(%eax) is FF 10, but this could also be part of direct call 0x10FF10FF. I.e.

12: ff 10                   call   *(%eax)
14: e8 fb 10 ff 10          call   10ff1114 <main+0x10ff1114>

E.g. if I find FF 10, then check if 3 bytes before there’s E8 – will it be enough or no? What other hidden traps that I did not think of?

Answer 1

Since you have slightly more information than usual, it’s slightly less impossible. But I’ll show exactly how it’s still impossible even with the extra information.

The extra information comes from knowing where it jumped to, and knowing the return address. If the difference doesn’t match the dword right before the return address then it wasn’t a direct call. So that’s something you can find out relatively easily, without having to mess with backwards decoding (which is not possible in general, only with luck and heuristics and code that wasn’t specifically crafted to defeat that, and even then a jump could have come in from anywhere at any place).

However, the code could be this:

  push returnaddr
  mov eax, calladdr
  jmp eax ;or whatever way you like, showing a silly jump through reg for no reason
  ...
  call calladdr
returnaddr:

And now it matches anyway, in a way that you can’t possibly detect.

So the only thing you can find out is whether it was definitely something other than a direct call. You can’t find out with certainty that any specific way was used – it is clearly possible to replace the call calladdr in the above snippet to make it look like any desired pattern.