Question

Our team is responsible for a large codebase containing legal rules.

The codebase works mostly like this:

class SNR_15_UNR extends Rule {
    public double getValue(RuleContext context) {
        double snr_15_ABK = context.getValue(SNR_15_ABK.class);
        double UNR = context.getValue(GLOBAL_UNR.class);
        if(UNR <= 0) // if UNR value would reduce snr, apply the reduction
          return snr_15_ABK + UNR;
        return snr_15_ABK;
    }
}

When context.getValue(Class<? extends Rule>) is called, it just evaluates the specific rule and returns the result. This allows you to create a dependency graph while a rule is evaluating, and also to detect cyclic dependencies.

There are about 500 rule classes like this. We now want to implement tests to verify the correctness of these rules.

Our goal is to implement a testing list as follows:

TEST org.project.rules.SNR_15_UNR
INPUT org.project.rules.SNR_15_ABK = 50
INPUT org.project.rules.UNR = 15
OUTPUT SHOULD BE 50

TEST org.project.rules.SNR_15_UNR
INPUT org.project.rules.SNR_15_ABK = 50
INPUT org.project.rules.UNR = -15
OUTPUT SHOULD BE 35

Question is: how many test scenario’s are needed? Is it possible to use static code analysis to detect how many unique code paths exist throughout the code? Does any such tool exist, or do I have to start mucking about with Eclipse JDT?

For clarity: I am not looking for code coverage tools. These tell me which code has been executed and which code was not. I want to estimate the development effort required to implement unit tests.

Answer 1

(EDIT 2/25, focused on test-coding effort):

You have 500 sub-classes, and each appears (based on your example with one conditional) to have 2 cases. I’d guess you need 500*2 tests.

If your code is not a regular as you imply, a conventional (branch) code coverage tool might not be the answer you think you want as starting place, but it might actually help you make an estimate. Code T<50 tests across randomly chosen classes, and collect code coverage data P (as a percentage) over whatever part of the code base you think needs testing (particularly your classes). Then you need roughly (1-P)*100*T tests.

If your extended classes are all as regular as you imply, you might consider generating them. If you trust the generation process, you might be able avoid writing the tests.

(ORIGINAL RESPONSE, focused on path coverage tools)

Most code coverage tools are “line” or “branch” coverage tools; they do not count unique paths through the code. At best they count basic blocks.

Path coverage tools do exist; people have built them for research demos, but commercial versions are relatively rare. You can find one at http://testingfaqs.org/t-eval.html#TCATPATH. I don’t think this one handles Java.

One of the issues is that the apparent paths through code is generally exponential in the number of decisions since each encountered decision generates a True path and a False path based on the outcome of the conditional (1 decision –> 2 paths, 2 decisions –> 4 paths, …). Worse loops are in effect a decision repeated as many times as the loop iterates; a loop that repeats a 100 times in effect has 2**100 paths. To control this problem, the more interesting path coverage tools try to determine the feasibility of a path: if the symbolically combined predicates from the conditionals in a prefix of that path are effectively false, the path is infeasible and can be ignored, since it can’t really occur. Another standard trick is treat loops as 0, 1, and N iterations to reduce the number of apparent paths. Managing the number of paths requires rather a lot of machinery, considerably above what most branch-coverage test tools need, which helps explain why real path coverage tools are rare.