Question

I’m new to using code coverage and not up on how they behave. The tools I have tried all seem to want to run the entire suite of unit tests to do their analysis. I had though that they would somehow analyze the code statically, that is, without having to run Unit Tests.

Is my assumption incorrect?

Answer 1

The usual open source tools, like Cobertura, Emma, etc., require running the tests. They use things like byte-code manipulation or AOP to instrument the code and track which lines get executed as the tests run, then produce reports of the results. The simplicity of it makes it a very attractive solution.

It’s really interesting that you can approximate test coverage with static analysis, but it seems like a last-ditch option for cases where running the tests is not practical. Quoting from the abstract from the paper quoted in Ira Baxter’s answer:

Test coverage is an important indicator for unit test quality. Tools
such as Clover compute coverage by first instrumenting the code with
logging functionality, and then logging which parts are executed
during unit test runs. Since computation of test coverage is a dynamic
analysis, it presupposes a working installation of the software. In
the context of software quality assessment by an independent third
party, a working installation is often not available. The evaluator
may not have access to the required libraries or hardware platform.
The installation procedure may not be automated or documented. In this
paper, we propose a technique for estimating test coverage at method
level through static analysis only. The technique uses slicing of
static call graphs to estimate the dynamic test coverage. We explain
the technique and its implementation. We validate the results of the
static estimation by statistical comparison to values obtained through
dynamic analysis using Clover. We found high correlation between
static coverage estimation and real coverage at system level but
closer analysis on package and class level reveals opportunities for
further improvement.