Question

What are some tips for debugging hard to reproduce concurrency bugs that only happen, say, once every thousand runs of a test? I have one of these and I have no idea how to go about debugging it. I can’t put print statements or debugger watches all over the place to observe internal state, because that would change timings and produce overwhelming amounts of information when the bug is not successfully reproduced.

Answer 1

Here is my technique : I generally use a lot of assert() to check the data consistency/validity as often as possible. When one assert fails, the program crashes generating a core file. Then I use a debugger with the core file to understand what thread configuration led to data corruption.