Question

I have an system running embedded linux and it is critical that it runs continuously. Basically it is a process for communicating to sensors and relaying that data to database and web client.

If a crash occurs, how do I restart the application automatically?

Also, there are several threads doing polling(eg sockets & uart communications). How do I ensure none of the threads get hung up or exit unexpectedly? Is there an easy to use watchdog that is threading friendly?

Answer 1

The gist of it is:

1. You need to detect if the program is still running and not hung.
2. You need to (re)start the program if the program is not running or is hung.

There are a number of different ways to do #1, but two that come to mind are:

1. Listening on a UNIX domain socket, to handle status requests. An external application can then inquire as to whether the application is still ok. If it gets no response within some timeout period, then it can be assumed that the application being queried has deadlocked or is dead.

2. Periodically touching a file with a preselected path. An external application can look a the timestamp for the file, and if it is stale, then it can assume that the appliation is dead or deadlocked.

With respect to #2, killing the previous PID and using fork+exec to launch a new process is typical. You might also consider making your application that runs “continuously”, into an application that runs once, but then use “cron” or some other application to continuously rerun that single-run application.

Unfortunately, watchdog timers and getting out of deadlock are non-trivial issues. I don’t know of any generic way to do it, and the few that I’ve seen are pretty ugly and not 100% bug-free. However, tsan can help detect potential deadlock scenarios and other threading issues with static analysis.