Question

Basically I want an arbitrarily large stack. I know that’s not possible, but could I set aside a few terabytes of my virtual address space for it? I’d like to be able to start at the beginning and walk up the buffer as far as I need, and Linux can bring in pages from physical memory on an as-needed basis. Is something like that possible? Would it have the same performance as just malloc-ing a buffer? Would there be a way to signal to Linux that you’re done with the memory once you pop the stack?

EDIT: I’m wanting this because I want to optimize a recursive/parallel algorithm that allocates lots of memory on each call. Malloc is too slow for me and I don’t want all the threads to trip on each other inside of malloc’s locks. So basically it would be my own run-time stack alongside the real one (one for each thread).

Actually, as long as the run-time stack is big enough that should be good enough. Is there a way to know/ensure the size of the stack? In a 64-bit address space there’s enough room for several threads to be stack-allocating gigabytes of data. Is that doable?

It looks like pthread_attr_setstacksize could work for new threads, but that doesn’t help much if you can be called from any thread …

Answer 1

You can create such an allocation using mmap() with the MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE flags. When you want to free it back to the system, use munmap().

Note that the only way you will find out if insufficient memory is available to satisfy your actual use is by your process receiving a SIGSEGV – but this is comes with the territory of asking for a giant mapping much larger than available memory.