Question

Suppose I have process which is performing some activity based on external conditions, which it can modify as time goes by, e.g. based on data stored in database. If such process is then forcefully killed and started again, the data will be inconsistent – e.g. information about service status will be inadequate or partially synchronized branches of data will be redundant.

How to handle such situation ? Should I check for some wired inconsistency at the booting of the process and normalize the data before the main logic starts, or can I detect the abort signal and make the cleaning then ? I’m not worrying about the resources acquired by the process, but uncompleted changes it has been already able to perform (which can be heavily detectable on the process fresh booting though).

Answer 1

Problems like this are generally solved in the following way:

1. Split up your process into several small actions
2. Make sure that each action is idempotent, meaning it can be executed multiple times without changing the result of the first execution
3. Advance the status of the whole process after an action completed successfully.

Like that, you always can re-start the process starting with the action that follows the last successfully executed one and go on from there.