I am working on a windows service that polls for a connection to a network enabled devices every 15 seconds. If the service is not able to connect to a device, it throws an exception and tries again in 15 seconds. All of this works great.
But, lets say one of the devices is down for a day or more. I am filling up my exception log with the same exception every 15 seconds. Is there a standard way to prevent an exception from being written to the event log if the exception being thrown hasn’t changed in the last x number of hours?
One good way to achieve what you need is to employ the Circuit Breaker design pattern.
I first read about this in the book ‘Release It! Design and Deploy Production Ready Software’ by Michael T. Nygard, from the Pragmatic Press, p104-107.
The idea of the circuit breaker is that it sits in the path of the connection between systems, passing connections through, watching for the ‘break condition’. For example, it might trigger only if five connections in a row have all failed.
Once the circuit has broken, all calls through the circuit breaker fail immediately, without consulting the external service. This continues until a timeout occurs, when the breaker goes into a half-open state. The next call is attempted – a failure results in the timeout being reset, success in the breaker closing and the system resuming operation.
A quick google found a post by Tim Ross that reads well and goes into more detail.
In your case, you could use a circuit breaker with a timeout of 10 minutes, and a trigger of 5 failures. Your log files would then contain, in the case of an all day failure, five exceptions logged for the original problem, and then just six more an hour (compared with 240 at 15 second intervals), indicating that the problem persists.
Depending on your requirements, you could include a manual ‘reset’ of the circuit breaker, or you could just leave it to automatically reset when the 10 minute timeout reveals things are back to normal. This could be useful – generally the fewer things the sysadmins need to fuss with, the better they like it.