Question

We are observing a core dump quite randomly, under heavy load conditions. When we load the core file and look at the location of the core dump it is always pointing to the last line of the function, precisely the line number of the closing brace.

The function has some legacy goto statements. When we had similar issue earlier, we moved creation of all local objects to the top of the function and that appeared to have fixed the issue on Solaris Unix 10. (Our suspicion and some sample tests showed that when goto statements were executed, some of these local variables were never created but their destructors were always invoked. So moving them all the way to the top ensured that they are always constructed properly). But the problem is still happening on the Linux, while we don’t see this issue any more on Solaris.

Updated with stack trace :

#0  0x008a5206 in raise () from /lib/libc.so.6

#1  0x008a6bd1 in abort () from /lib/libc.so.6

#2  0x008de3bb in __libc_message () from /lib/libc.so.6

#3  0x00966634 in __stack_chk_fail () from /lib/libc.so.6

#4  0x08e9ebf5 in our_function (this=0xd2f2c380)

    at sourcefilename.cc:9887

Anybody encountered similar issue? Greatly appreciate any help or pointers to understand and fix the issue. Thanks a ton.

Answer 1

I suspect you’re overrunning a buffer in a growing-downwards stack (most stacks grow downwards; I don’t know whether Linux or Solaris use downwards stacks on all architectures, but definitely some of them). At this point, it overwrites the return address, and the program counter jumps to an illegal address, generating the crash at precisely where the function returns.

Just Valgrind it, it will probably tell you what’s happening (or rather, where the overrun is).