Question

I am writing unit tests for our main product and have one concern: how to differentiate

• tests that failed because what they tested went wrong (a bug was found and its a non regression test for it for instance)
• tests that failed because another unexpected part of the test failed (because the test is wrong or an unknown bug appeared)

For the first one, we have the jUnit Assert framework for sure, but what do we have for the second one?

Example: my unit test is testing that c() will not throw MyException, but to perform c() I need to first perform a() then b() which both can throw MyException(), so I would write:

@Test
public void testC() {
  a();
  Object forC = b();
  try {
    c(forC);
  } catch (MyException e) {
    Assert.fail("....");
  }
}

But then I need to handle the MyException that can be thrown by a or b, and also handle the fact that forC should not be null. What is the best way to do this?

• Catch MyException thrown by a or b and Assert.fail, but a and b are not tested by this test so for me they shouldn’t be marked as test failure when they fail. Maybe they fail later because at this time we should do b();a() not a();b();.
• Let testC throws MyException, so that the test will fail with “MyException”, but this is misleading because MyException will not tell that the test is wrongly written. All tests will then fail each with their own Exception. In this case I would also need to throw something like NullPointerException if forC is null, which also has no semantic.
• Catch and wrap MyException thrown by a and b into a exception telling that the test might be wrong, like a TestCorruptedException. But I couldnt find such exceptions in jUnit, so they would not be recognized by jUnit as such (that’s ok for me). Also I would need to know this exception from all my unit tests which are of course split into multiple modules, projects and so on…So this is doable but adds dependencies.

What would be your solution to this problem? I will likely go for the second but I’m not pleased with it as explained above.

Answer 1

The corner stone of unit testing is to test the minimum amount of code necessary, otherwise it arguably falls into the integration test space where you are looking for end-to-end functionality.

If you can prove that a() can be created and tested in it’s own test class, and that b() can also be created and tested in it’s own test class, then it follows that your test above can ignore the testing of a() and b() and instead use known values that won’t fail. This is commonly satisfied by using mock objects.

With a() and b() created as mock objects, c() can be tested in isolation. If you find that it is not possible to test c() in isolation of a() and b() then this is an indicator that your code needs to change so that there is a separation of concerns. This is commonly satisfied by injecting the dependany of a() and b() into c().

This post on when to use mock objects in unit testing may help shed some more light on this topic.