Greetings,
I am currently exploring some extreme programming and try to stick as much to it as possible. This means, I will need to turn my (by now, unexpectedly thick stack of) user stories into acceptance tests once I begin an iteration (after planning the release, of course).
I am not entirely sure about the implementation language I am going to use, however, I am sure that this is going to be a dynamic web application with a database backend, served by a webserver. Right now, I plan to develop the first release on a local machine with a local testing environment, so it is possible to assume that security is no concern on the acceptance tests (so, I can give the acceptance tests root access to the testing database involved, for example). I am still a bit unsure about the acceptance test framework to use, however, since this is going to be a web application, I think I will use Selenium RC in order to write the tests and run them (I mention this in case someone is able to point me to something better 🙂 ).
However, there still is a dark area left: I do not have data for this application yet, because I am implementing a new, fresh application. Thus, I cannot grab a snapshot of the current production database in order to grab a test database, and additionally, the application is stateful (as any web application with a database backend is), so using a single database for all acceptance tests is going to cause ugly problems with regard to test isolation (and at least for unit tests, that reads as “This can result in great fun and lots of gray hair”).
So, how to I solve this problem? Do I create artificial testing databases (and maintain them whenever the database schema changes) and write the acceptance tests such that each acceptance test loads the appropiate database state into the testing database before running the test? (How fast or slow will it be to load a dozen records a hundred times, when a lot of accentance tests run?) Should I create a single example database, load this for all tests and hope for the best? Should I recreate the test data I need all the time in the acceptance tests? Or, how do people do this?
According to further research, the proper way to do this is to bring the database into a defined state using the appropiate setUp-methods. This porentially involves deleting all existing data in the tables, adding a certain test set of data to the table and then running the test on exaclty this data. Afterwards, the teardown-method clears up whatever was done to the tables (either setUp drops everything, or teardown drops everything again). There are tools like dbUnit to simplify this process. This results in some reduction of the testing speed, however, it establishes total isolation of tests, which is a good thing, because then, green simply means green and red simply means red and not “Given the current order of test execution, this works”.
Besides that, the speed issue will probably be less significant for me, as I can focus on a small amount of tests during developing code for a single user story and have my CI-server run all the tests (which then takes more time) in the background when I think I am done.