We have a huge ASP.NET web application which needs to be deployed to LIVE with zero or nearly zero downtime. Let me point out that I’ve read the following question/answers but unfortunately it doesn’t solve our problems as our architecture is a little bit more complicated.
Let’s say that currently we have two IIS servers responding to requests and both are connected to the same MSSQL server. The solution seems like a piece of cake but it isn’t because of the major schema changes we have to apply from time to time. Because of it’s huge size, a simple database backup takes around 8 minutes which has become unacceptable, but it is a must before every new deploy for security reasons.
I would like to ask your help to get this deployment time down as much as possible. If you have any great ideas for a different architecture or maybe you’ve used tools which can help us here then please do not be shy and share the info.
Currently the best idea we came up is buying another SQL server which would be set up as a replica of the original DB. From the load balancer we would route all new traffic to one of the two IIS webservers. When the second webserver is free of running sessions then we can make deploy the new code. Now comes the hard part. At this point we would go offline with the website, take down the replication between the two SQL servers so we directly have a snapshot of the database in a hopefully consistent state (saves us 7.5 of the 8 minutes). Finally we would update the database schema on the main SQL server, and route all traffic via the updated webserver while we are upgrading the second webserver to the new version.
Please also share your thoughts regarding this solution. Can we somehow manage to eliminate the need for going offline with the website? How do bluechip companies with mammuth web applications do deployment?
Every idea or suggestion is more than welcome! Buying new hardware or software is really not a problem – we just miss the breaking idea. Thanks in advance for your help!
Edit 1 (2010.01.12):
Another requirement is to eliminate manual intervention, so in fact we are looking for a way which can be applied in an automated way.
Let me just remind you the requirement list:
1. Backup of database
2a. Deploy of website
2b. Update of database schema
3. Change to updated website
4 (optional): easy way of reverting to the old website if something goes very wrong.
First off, you are likely unaware of the “point in time restore” concept. The long and short of it is that if you’re properly backing up your transaction logs, it doesn’t matter how long your backups take — you always have the ability to restore back to any point in time. You just restore your last backup and reapply the transaction logs since then, and you can get a restore right up to the point of deployment.
What I would tend to recommend would be reinstalling the website on a different Web Site definition with a “dead” host header configured — this is your staging site. Make a script which runs your db changes all at once (in a transaction) and then flips the host headers between the live site and the staging site.