This question is language independent.
I have an application that handles requests in a loop. During this loop for each request multiple actions are taken. These actions are sitting inside try / catch / log blocks.
I am now extending this to notify administrators of severe errors via email.
This is all very easy, except for one thing. We are relying on the client/s to implement their own email delivery redundancy, and I know from experience there will always be one client who just has one SMTP exchange server, and this is bound to go down from time to time.
So here is the dilemma:
Scenario 1 (don’t handle the error during failed send) – when I send an email to admin and SMTP is down it will break the app (app will stop running, and additional loops will stop processing, because the error is unhandled) This means that the error reporting which was supposed to be beneficial to the app suddenly becomes the reason why 99/100 requests don’t get processed because there was an issue with request 1.
Scenario 2 (handle the exception during failed send) – this means that I surround send code in try/catch/log blocks, great! the application processes all requests 99 of them, except one, but the admin now has no notification of this one error via email because when it tried to send SMTP was down, and that error was simply logged to the application log, the admin who doesn’t check this log for days (even weeks) at a time now has no way to know that error took place.
So is there a win/win way to solve this problem or am I always going to be at a loss, and in the mercy of SMTP being up. Remember it is out of our scope to manage email server redundancy.
Extend scenario 2 to keep a record of which entries in the application log didn’t get sent via email, and periodically poll this log for unsent entries and try to resend them – eventually the smtp service will be available again. (You might want to stop any resent errors from going back in the resend queue tho…)