I have a Windows service that monitors several mailboxes for new mail. Once new mail arrives the service is responsible for determining what to do with it. It can either parse it and enter it into a database, ignore it, or forward it to someone if it doesn’t recognize it.
I already have a working implementation if/when our email server goes down where the service will try and reconnect every n minutes until successful. The other problem is what happens if/when the database server goes down. Our service could very well still be running and parsing email but would stall if the database insertion fails.
How do I handle the database going down? Instead of having my service responsible for interacting to the database do I use a messaging framework like NServiceBus and send messages for each new item that needs to get added. Then this service would try and process it. If the processing fails for whatever reason (most likely the database is down) simply leave the item in the queue to process for later?
One possibility:
If the database is down, serialize the completed work to XML or binary format and store it on the file system if the database is down. Consider this the offline queue. When the database comes back up, have the process de-serialize the data back and then store it in the database.
That way you can still consider the data processed and insert it at a later time. The queue (back up) is the file system.