Question

I am trying to understand how stack allocation and alignment works with pthreads on a Linux x86_64 system with gcc and what data the system stores on the stack. I know that you can configure the stack memory using pthread_attr_setstack. I’ve done this in a test program that does the following:

1) recursively calls itself and updates an uninitialized array that is allocated on the stack

2) prints out the value of the first array element, the last element, and rsp

From this I’ve been able to observe how rsp gets incremented (in my test program I noticed some of the recursive calls were inlined by the compiler). I’ve also been able to see that adding TLS memory (with __thread variables) causes the first value of rsp to be lower. So it looks like TLS variables are allocated on the top of the stack.

However, what I’m not sure about is what else is there. It looks to me like the first page of the stack is reserved for the system in some way because none of the stack variables that I allocate end up in that region. Even if I don’t use any __thread variables the variables that I do instantiate do not appear to be allocated in the first page (I set the stack memory so that it is page aligned).

So my question is: what else, if anything, is on the stack for a pthread besides TLS data and stack variables?

Answer 1

On Linux NPTL:

The very top of the stack contains the TCB. This is also known as struct pthread or pthread_t. It’s a bit hairy due to all the weird glibc system-specific defines, but basically it contains things like:

• The TLS header (this part does not include actual TLS values)
• Various flags
• A linked list for tracking cached stacks
• TID (and stack in-use flag)
• PID
• Robust mutex tracking information
• pthread_cleanup and stack unwind information
• Cancellation state
• Even more flags
• A certain number of TLS slots used by pthread_setspecific – if you exceed this number, the rest are allocated on the heap
• Various locks
• Tracking information for pthread_join
• The thread’s return value
• Scheduling policies (from the thread attributes)
• The start routine and thread argument
• Stack size tracking data
• etc etc

This is mostly initialized in pthread_create (__pthread_create_2_1 in nptl/pthread_create.c if you’re following along in the eglibc source), before the new thread actually starts running.

Below the TCB is statically allocated __thread variables – or, at least, those the linker could identify at startup time. The dynamic linker initializes a l_tls_offset field in the linker map to tell the NPTL code how much space to reserve. Note that libraries loaded after program startup won’t be part of this – see the __thread ABI spec for details.

Below the __thread variables is the stack. The top of this stack is the start_thread() code, so it’ll still be a ways before it actually executes user code (but not too much further).