Question

I am trying to set up Cobertura code coverage on a project which includes auto-generated code, created from templates using Freemarker (explicitly, using the fmpp maven plugin).

I then have unit tests for those auto-generated classes.

However, these unit tests are not being considered by Cobertura when calculating code coverage. The auto-generated classes appear in the report, but the coverage on those classes is always 0%.

First, is there some configuration for Cobertura that I’m missing?

This SO question appears to have been asking a similar question, but the accepted answer is that:

Generated code should not be tested and should not be used in code coverage metrics.

This does not seem right to me – I think I should be testing the generated code (both to test the generated classes and the templates), and I want to know how the code coverage for this generated code.

So, second, is there a good reason why generated code shouldn’t be tested?

Edit: It’s also relevant to mention that I am using cobertura using the cobertura-maven-plugin. As such, I’m not sure if the problem is with cobertura or the maven-plugin (or my configuration thereof…)

N.b. to be clear, I am not asking about auto-generating the test classes. These are manually written, to test the classes created from templates.

Answer 1

This is a partial answer to report what I’ve learnt so far…

First, I’ve had a look at how cobertura works, and it seems to be as follows:

• the project java classes are amended to include calls to TouchCollector.touch(…) (and similar methods) throughout the code, and compiled. (This is the instrumentation task.)
• the tests are run on these cobertura compiled classes, and TouchCollector records which lines of the project classes have been reached in the tests. This information is recorded in the cobertura.ser data file.
• the cobertura / report task reads the cobertura.ser data file and produces the html report.

So, my first thought was that the auto-generated classes were not being instrumented properly. However, after decompiling these cobertura class files, I can confirm that both normal and auto-generated classes have been instrumented correctly.

Also, I can confirm that all the tests – including the tests of the auto-generated classes are being run.

Then, my next question is why, when the tests are run, the auto-generated classes are not being ‘touched’. Further investigation showed that when the test classes were being compiled (i.e. test-compile), the auto-generated classes are added to project/target/test-classes. A simple test to print the file location of an auto-generated class (e.g. System.out.println(MyAutoClass.class.getResource("MyAutoClass.class"));) confirmed that when the tests are run by cobertura, it uses the auto-generated classes in the test-classes folder, and not the cobertura compiled classes that had been instrumented.

Then, the next question is how to prevent these classes being added to the test-classes folder…? Well, one approach is to exclude the auto-generated classes from being compiled. This can be done by only including the Test classes – e.g.:

<plugin>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>x.y.z</version>
  <configuration>
    <testIncludes>
      <testInclude>**/*Test.java</testInclude>
    </testIncludes>
  </configuration>
</plugin>

Or it should be possible to exclude the auto-generated classes (which would be preferred) – something along the lines of:

<testExcludes>
  <testExclude>**/generated-sources/fmpp/**/*.java</testExclude>
</testExcludes>

However, this didn’t work, and I’m not quite sure how to get this to work.

An alternative could be to move all the auto-generated classes into a single package, and then something like the following could be possible:

<testExcludes>
  <testExclude>com/organisation/project/auto/**/*.java</testExclude>
</testExcludes>

In the end, I went for the only including *Test.java files