Question

I have divided the whole question into smaller ones:

1. What kind of different algorithms GDB is capable to use to reconstruct stacktraces?
2. How each of the stacktrace reconstruction algorithm works at high level? Advantages and disadvantages?
3. What kind of meta-information compiler needs to provide in program for each stacktrace reconstruction algorithm to work?
4. And also corresponding g++ compiler switches that enable/disable particular algorithm?

Answer 1

Speaking Pseudocode, you could call the stack "an array of packed stack frames", where every stack frame is a data structure of variable size you could express like:

template struct stackframe<N> {
    uintptr_t contents[N];
#ifndef OMIT_FRAME_POINTER
    struct stackframe<> *nextfp;
#endif
    void *retaddr;
};

Problem is that every function has a different <N> – frame sizes vary.

The compiler knows frame sizes, and if creating debugging information will usually emit these as part of that. All the debugger then needs to do is to locate the last program counter, look up the function in the symbol table, then use that name to look up the framesize in the debugging information. Add that to the stackpointer and you get to the beginning of the next frame.

If using this method you don’t require frame linkage, and backtracing will work just fine even if you use -fomit-frame-pointer. On the other hand, if you have frame linkage, then iterating the stack is just following a linked list – because every framepointer in a new stackframe is initialized by the function prologue code to point to the previous one.

If you have neither frame size information nor framepointers, but still a symbol table, then you can also perform backtracing by a bit of reverse engineering to calculate the framesizes from the actual binary. Start with the program counter, look up the function it belongs to in the symbol table, and then disassemble the function from the start. Isolate all operations between the beginning of the function and the program counter that actually modify the stackpointer (write anything to the stack and/or allocate stackspace). That calculates the frame size for the current function, so subtract that from the stackpointer, and you should (on most architectures) find the last word written to the stack before the function was entered – which is usually the return address into the caller. Re-iterate as necessary.

Finally, you can perform a heuristic analysis of the contents of the stack – isolate all words in the stack that are within executably-mapped segments of the process address space (and thereby could be function offsets aka return addresses), and play a what-if game looking up the memory, disassembling the instruction there and see if it actually is a call instruction of sort, if so whether that really called the ‘next’ and if you can construct an uninterrupted call sequence from that. This works to a degree even if the binary is completely stripped (although all you could get in that case is a list of return addresses). I don’t think GDB employs this technique, but some embedded lowlevel debuggers do. On x86, due to the varying instruction lengths, this is terribly difficult to do because you can’t easily "step back" through an instruction stream, but on RISC, where instruction lengths are fixed, e.g. on ARM, this is much simpler.

There are some holes that make simple or even complex/exhaustive implementations of these algorithms fall over sometimes, like tail-recursive functions, inlined code, and so on. The gdb sourcecode might give you some more ideas:

https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/frame.c

GDB employs a variety of such techniques.