Question

Is there a way to document testNG testcases so that the documentation can be automatically generated? I would imagine something like:

@Description("This testcase checks, whether the system is alive")
@Step("Check whether services are running")
@Step("Try to access the system webpage")
...
@Test(groups={"sanity"})
public void checkStatus() {
...
}

I have considered two options: custom annotation processing and writing my own doclet for javadoc. Before I try any of these options, I would like to know, whether there is a standard way of doing it. I would definitely like to reuse current testNG annotations, especially the grouping of the tests.

Last but not least, I would like to mention that I want to use this approach only for system level tests (NOT unit tests), which I quite complex and where it is not so easy to say what the test does just from the test name or its code.

Answer 1

Finally, I figured it out myself. I use javadoc combined with a little bit of annotation processing (just to distinguish the testcase groups). I use a custom doclet, which first builds the list of testcases in the following way:

private MethodDoc[] getTestMethods(RootDoc root) {
    List<MethodDoc> result = new ArrayList<MethodDoc>();
    for (ClassDoc c : root.classes()) {
        for(MethodDoc m : c.methods()) {
            if (isTest(m))
                result.add(m);
        }
    }
    return result.toArray(new MethodDoc[0]);
}

// simplified test detection
private static boolean isTest(MethodDoc m) {
    for(AnnotationDesc a: m.annotations()) {
        if (a.annotationType().name().equals("Test"))
            return true;
    }
    return false;
}

Then, for each test I retrieve the set of groups:

static Set<String> getMethodGroups(MethodDoc m) {
    Set<String> result = getGroups(m);
    result.addAll(getGroups(m.containingClass()));
    return result;
}

static Set<String> getGroups(ProgramElementDoc m) {
    Set<String> result = new HashSet<String>();
    for(AnnotationDesc a: m.annotations()) {
        if (a.annotationType().name().equals("Test")) {
            for(ElementValuePair ev : a.elementValues()) {
                if (ev.element().name().equals("groups")) {
                    String value = ev.value().toString();
                    StringTokenizer tokenizer = new StringTokenizer(value, "{}\", ");
                    while (tokenizer.hasMoreElements()) {
                        result.add(tokenizer.nextToken());
                    }
                }
            }
        }
    }
    return result;
}

The rest is just standard doclet processing. In addition, I found out that I can use custom tags directly in javadoc, e.g.

/**
 * Some description
 * @Step Step 1
 * @Step Step 2
*/
void someMethod() {}