Recently we added a couple web service machines, and they couldn’t successfully email out. We (IS) did not notice this, and the exceptions were being swallowed up and logged, but no-one noticed for about a month.
Needless to say, many purchase orders, and retraction of purchase orders, were never sent out for the past month.
While this isn’t any one person’s fault really, is there any GOOD way to break this to someone non-technical that is higher up in the company than you?
Thanks in advance for any advice, I’m freaking out just a bit. 🙂
Edit: Reading this over, I’m more asking for tips on how to break the news. I understand there isn’t a GOOD way, just maybe successful tips that have worked for you in the past.
Resolution in case anybody was wondering… new web service machines IPs weren’t added to our mail servers list of trusted IPs. 🙂
Put emphasis on the fact that the problem was discovered and fixed swiftly by your team. Have detailed metrics on the number of failures, which customers were affected, etc. ready, in-hand. Have a contingency plan ready to describe that will prevent similar issues from happening in the future. Engender a sense of comradery with the higher-up because you are all on the same team and it’s a team problem. If you convey a sense of urgency and give the impression that you appreciate the impact to the bottom line as much as they do, they will respond much better.
Lowly techs often make the mistake of going to upper management with their tail tucked between their legs, like a child who shamefully shows his parents the lamp he broke and waits for a spanking. You are an adult and a professional – leap into action and coordinate the right people to be in place to make the right decisions to fix it. In a case like this, that inevitably means bringing in upper management, but do so with an intention of solution seeking, not fear.