I’m working with a somewhat unreliable (Qt/windows) application partly written for us by a third party (just trying to shift the blame there). Their latest version is more stable. Sort of. We’re getting fewer reports of crashes, but we’re getting lots of reports of it just hanging and never coming back. The circumstances are varied, and with the little information we can gather, we haven’t been able to reproduce the problems.
So ideally, I’d like to create some sort of watchdog which notices that the application has locked up, and offers to send a crash report back to us. Nice idea, but there are problems:
-
How does the watchdog know the process has hung? Presumably we instrument the application to periodically say ‘all ok’ to the watchdog, but where do we put that such that it’s guarenteed to happen frequently enough, but isn’t likely to be on a code path that the app ends up on when it’s locked.
-
What information should the watchdog report when a crash happens? Windows has a decent debug api, so I’m confident that all the interesting data is accessible, but I’m not sure what would be useful for tracking down the problems.
You want a combination of a minidump (use DrWatson to create these if you don’t want to add your own mini-dump generation code) and userdump to trigger a minidump creation on a hang.
The thing about automatically detecting a hang is that its difficult to decide when somethings hung and when its just slow or blocked by IO wait. I personally prefer to allow the user to crash the app deliberately when they think its hung. Apart from being a lot easier (my apps don’t tend to hang often, if at all 🙂 ), it also helps them to ‘be part of the solution’. They like that.
Firstly, check out the classic bugslayer article concerning crashdumps and symbols, which also has some excellent information regarding what’s going on with these things.
Second, get userdump which allows you to create the dumps, and instructions for setting it up to generate dumps
When you have the dump, open it in WinDBG, and you will be able to inspect the entire program state – including threads and callstacks, registers, memory and parameters to functions. I think you’ll be particularly interested in using the ‘~*kp‘ command in Windbg to get the callstack of every thread, and the ‘!locks’ command to show all locking objects. I think you’ll find that the hang will be due to a deadlock of synchronisation objects, which will be difficult to track down as all threads tend to wait on a WaitForSingleObject call, but look further down the callstacks to see the application threads (rather than ‘framework’ threads like background notifications and network routines). Once you’ve narrowed them down, you can see what calls were being made, possibly add some logging instrumentation to the app to try and give you more information ready for the next time it fails.
Good luck.
Ps. Quick google reminded me of this: Debugging deadlocks. (CDB is the command line equivalent of windbg)