Question

What free tools can I use to debug multithreaded programs created using the pthread library in Linux? (Besides pen & paper, of course…)

The usual way of printing debug messages isn’t working out very well.

Answer 1

Debugging concurrent programs is inherently difficult, because the debugging tool tends to change the scheduling (often making it tamer so that the bug disappears).

One technique I’ve had some success with is to log to a data structure that is not protected by a lock. Then, once the system is idle, print the data structure or look at it in a debugger. The important thing is to avoid making a system call or invoking a synchronization primitive when logging, so that logging has minimal influence on the scheduler.

static char *log_buffer[LOG_BUFFER_LENGTH];
static size_t log_index;
#define LOG(message) (log_buffer[log_index++] = (message))

If your thread is interrupted in the middle of logging, the log buffer will become inconsistent. This is improbable enough to be useful for debugging, though it must be kept in mind. I’ve never tried this on a multiprocessor machine; I would expect the lack of synchronization to make the log buffer inconsistent very quickly in practice¹.

¹
_{Which is one more reason not to do multithreaded programming on multiprocessor machines. Use message passing instead.}