I’m principally interested in case studies on code metrics, relating code readability to defect reduction, that justify taking seriously cyclomatic complexity or some similar metric. Wikipedia has this example:
A number of studies have investigated
cyclomatic complexity’s correlation to
the number of defects contained in a
module. Most such studies find a
strong positive correlation between
cyclomatic complexity and defects:
modules that have the highest
complexity tend to also contain the
most defects. For example, a 2008
study by metric-monitoring software
supplier Enerjy analyzed classes of
open-source Java applications and
divided them into two sets based on
how commonly faults were found in
them. They found strong correlation
between cyclomatic complexity and
their faultiness, with classes with a
combined complexity of 11 having a
probability of being fault-prone of
just 0.28, rising to 0.98 for classes
with a complexity of 74.
This is good, but I’m hoping to know if there are more studies (or perhaps similar studies for other metrics, such as SLOC).
I also found an article at IBM that promotes monitoring CC values, but it lacks case-study support showing ROI figures. Then there is Coding Horror article on “arrow code” which sites a summary of a case study, but does not offer the case study(ies) themselves nor the actual numbers which justified the conclusion:
Studies show a correlation between a
program’s cyclomatic complexity and
its error frequency. A low cyclomatic
complexity contributes to a program’s
understandability and indicates it is
amenable to modification at lower risk
than a more complex program. A
module’s cyclomatic complexity is also
a strong indicator of its testability.
Certainly cyclomatic complexity (CC) will help spot arrow-code, but I still need case studies that show ROI values. For example, “organization X incorporated a max CC of 10 on methods/functions, and reduced defects 20% in the following development iteration.”
Without that kind of data, it is difficult to get management to care. Can anyone point me to a few hard studies? Even just one would help…
In this case, you get a bit more from the original article than Wikipedia’s. His technical paper about how the data gathering and such shows a 95% confidence level in the conclusions.
You’re right that this doesn’t give ROI information directly. At least for this study, that would be fairly difficult — for example, they used open-source projects for their training data, and actual costs for open-source projects are usually difficult to even estimate, much less measure. At the same time, they did use what I’d consider at least a reasonable proxy for true ROI data: they searched through the source control system for each of their “training” projects looking for check-ins that appeared to be related to fixing bugs, defects, etc. They then used a naive Bayes algorithm to find correlation between the metrics they used, and the problems that had been identified in the code. While undoubtedly open to at least some improvement, it appears to me that these results should mean at least something.
It’s also worth noting that the same people who did the study keep a running index on a large number of open-source projects. If you wanted more in the way of solid data, you can check their index back against the source control logs for some of those projects, you could probably use their data to come up with more in the way of direct ROI type results. One note, however: their index is based on quite a few source code metrics, not just cyclomatic complexity, so I’m not sure exactly how much it would tell exclusively about CC as opposed to the other metrics they look at.