I cannot believe I’m the first person to go through this thought process, so I’m wondering if anyone can help me out with it.
Current situation: developers write a web site, operations deploy it. Once deployed, a developer Smoke Tests it, to make sure the deployment went smoothly.
To me this feels wrong, it essentially means it takes two people to deploy an application; in our case those two people are on opposite sides of the planet and timezones come into play, causing havoc. But the fact remains that developers know what the minimum set of tests is and that may change over time (particularly for the web service portion of our app). Operations, with all due respect to them (and they would say this themselves), are button-pushers who need a set of instructions to follow.
The manual solution is that we document the test cases and operations follow that document each time they deploy. That sounds painful, plus they may be deploying different versions to different environments (specifically UAT and Production) and may need a different set of instructions for each.
On top of this, one of our near-future plans is to have an automated daily deploy environment, so then we’ll have to instruct a computer as to how to deploy a given version of our app. I would dearly like to add to that instructions for how to smoke test the app.
Now developers are better at documenting instructions for computers than they are for people, so the obvious solution seems to be to use a combination of nUnit (I know these aren’t unit tests per se, but it is a built-for-purpose test runner) and either the Watin or Selenium APIs to run through the obvious browser steps and call to the web service and explain to the Operations guys how to run those unit tests. I can do that; I have mostly done it already.
But wouldn’t it be nice if I could make that process simpler still?
At this point, the Operations guys and the computer are going to have to know which set of tests relate to which version of the app and tell the nUnit runner which base URL it should point to (say, http://www.example.com = v3.2 or test.example.com = v3.3).
Wouldn’t it be nicer if the test runner itself had a way of giving it a base URL and letting it download say a zip file, unpack it and edit a configuration file automatically before running any test fixtures it found in there?
Is there an open source app that would do that? Is there a need for one? Is there a solution using something other than nUnit, maybe Fitnesse?
For the record, I’m looking at .NET-based tools first because most of the developers are primarily .NET developers, but we’re not married to it. If such a tool exists using other languages to write the tests, we’ll happily adapt, as long as there is a test runner that works on Windows.
After much time wasted trying to make up an easier solution, we eventually tought the ops team how to use NUnit’s Gui runner. This was easier than expected and is working fine.