As shown in a domain-manager article, I am interested in creating an integration test harness that creates a server and many clients, where all of them are running within a single process. But once I achieve this goal I will be very tempted to execute such integration & system tests from within a unit testing framework (NUnit or VS Team test).
What are the pros and cons of choosing to run integration or system tests from within a unit test framework? Here is my own stab:
Pros
- It can reduce time and cost of performing integration tests.
- Integration tests are achieved possibly with every build.
Cons
- Not all the integration tests will be fast. Some of them make take a minute to run (such as a performance-oriented integration test).
Either way, if my new integration testing code is not placed into a unit testing framework, where would you recommend it be placed? Said differently, what "integration test" frameworks would you recommend?
I don’t think there is anything wrong with writing your integration tests using the Unit Testing Framework you are already familiar with. Since these are integration tests there are a couple of things that differentiate them from your unit tests. One is the fact that they are dependent on external systems (even if you are spinning some of them up in your code), so if one of those is not available, your tests will fail. The other, as you have already pointed out, is the fact that integration tests take longer to run.
The way to deal with these issues is to configure your build scripts to not run the integration tests during your CI builds, but run them instead during a scheduled build (such as a nightly build). It is also important that the developers be able to run the integration tests manually on-demand, so that they can verify on their local machines that none of the tests have broken, and if they have, be able to verify that they have corrected the problem without manually triggering a system build.
Depending on the testing framework your using there are different ways of separating your unit tests from your integration tests so that you can configure your builds to execute one or both. One way is to move your integration tests into a separate project, and only execute tests in that project during your nightly build. The other is to use something like the Category attribute in NUnit to mark some tests as integration tests. You can then configure the test runner to exclude tests in this category for the builds that you do not want to execute your integration tests.