Question

How does one detect whether a segfault is being caused by an out-of-memory condition?

I have a segfault that defies diagnosis by valgrind and duma/efence because it seems to crash those tools themselves (Valgrind “the impossibe happened”, duma: “mprotect() failed: Cannot allocate memory” )

The application (Gazebo) simply crashes with a segfault, and a stack trace that doesn’t seem to offer many hints as to why.

TLDR: Is there an easy tool or method to either confirm or rule out the out-of-memory condition is the cause of a segfault?

(top does not show an inordinate amount of memory use before crash)

Answer 1

On Linux, an out-of-memory condition can manifest in one of two ways:

• If overcommit is disabled, a brk() or mmap() call fails with ENOMEM. Shortly thereafter, the application attempts to dereference the NULL pointer returned from malloc() and crashes.
• If overcommit is enabled, then the OOM killer kicks in, and kills the process with SIGKILL. A message is left in dmesg.

As such, you can rule out OOMs by checking that strace doesn’t show brk() or mmap() calls failing with ENOMEM, and verifying that no OOM killer messages appear in dmesg.