Question

For a customer we need to generate detailed test reports for integration tests which not only show, that everything is green, but also what the test did. My colleagues and I are lazy guys and we do not want to hack spreadsheets or text documents.

For that, I think about a way to document the more complex integration tests with JavaDoc comments on each @Test annotated method and each test class. For the test guys it is a good help to see to which requirement, Jira ticket or whatever the test is linked to and what the test actually tries to do. We want to provide this information to our customer, too.

The big question now is: How can we put the JavaDoc for each method and each test class into the JUnit reports? We use JUnit 4.9 and Maven.

I know, that there is a description for each assertXXX(), but we really would need a nice HTML list as result or a PDF document which lists all classes and there documentation and below that all @Test methods and their description, the testing time, the result and if failed, the reason why.

Or is there another alternative to generate fancy test scripts? (Or should we start an OpenSource project on this!? 😉 )

Update:
I asked another question on how to add a RunListener to Eclipse to have it also report in Eclipse when started there. The proposed solution with a custom TestRunner is another possibility to have the test results report. Have a look: How can I use a JUnit RunListener in Eclipse?

Answer 1

One way to achieve this would be to use a custom RunListener, with the caveat that it would be easier to use an annotation rather than javadoc. You would need to have a custom annotation such as:

@TestDoc(text="tests for XXX-342, fixes customer issue blahblah")
@Test
public void testForReallyBigThings() {
    // stuff
}

RunListener listens to test events, such as test start, test end, test failure, test success etc.

public class RunListener {
    public void testRunStarted(Description description) throws Exception {}
    public void testRunFinished(Result result) throws Exception {}
    public void testStarted(Description description) throws Exception {}
    public void testFinished(Description description) throws Exception {}
    public void testFailure(Failure failure) throws Exception {}
    public void testAssumptionFailure(Failure failure) {}
    public void testIgnored(Description description) throws Exception {}
}

Description contains the list of annotations applied to the test method, so using the example above you can get the Annotation TestDoc using:

description.getAnnotation(TestDoc.class);

and extract the text as normal.

You can then use the RunListener to generate the files you want, with the text specific to this test, whether the test passed or failed, or was ignored, the time taken etc. This would be your custom report.

Then, in surefire, you can specify a custom listener, using:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.10</version>
  <configuration>
    <properties>
      <property>
        <name>listener</name>
        <value>com.mycompany.MyResultListener,com.mycompany.MyResultListener2</value>
      </property>
  </configuration>
</plugin>

This is from Maven Surefire Plugin, Using JUnit, Using custom listeners and reporters

This solution has the disadvantage that you don’t have the flexibility of javadoc as far as carriage returns, formatting is concerned, but it does have the advantage that the documentation is in one specific place, the annotation TestDoc.