On many operating systems the stack and heap begin at opposite sides of a process’s virtual address space and grow toward one another. This allows the stack to expand as much as possible without hitting the heap.
Suppose that I have a program that causes a stack overflow. My current understanding is that this would result in the stack growing uncontrollably toward the heap and eventually hitting it. Is this correct? If it is, how does the operating system detect that a stack overflow occurs? It seems like the OS wouldn’t be able to detect that the program was trying to use virtual memory allocated for the heap as part of the stack, since they’d be in contiguous memory regions.
I know that this is operating-system specific, but insight on the mechanism by which this occurs in any operating system would definitely be helpful. This has been bugging me for a while and I can’t seem to find any good explanations.
The OS allocates some space to the stack. When the process accesses an unallocated part of the stack, a page fault is raised by the processor and caught by the OS. If the OS believes it’s still reasonable to grow the stack, it simply allocates new space for it and returns control to the process. If it’s not reasonable, a stack overflow exception is raised.