For many kinds of hard computational problems e.g. propositional satisfiability, theorem proving, traveling salesman etc., there exists an extensive test suite that can be used for evaluating the performance of a program that attempts to solve the problem.
Does any such thing exist for automatic bug finding, i.e. a collection of programs or code fragments containing known bugs of a kind that could be automatically detected? I assume e.g. Coverity must have such a thing for internal use, but a Google search doesn’t seem to show anything publicly available.
I don’t think such a suite exists. The most glaring question for a “standard” suite is, what language would it be written in?
For C, you can look at SAFECode‘s test suite, here. SAFECode is meant to detect these bugs at runtime, but a “perfect” static analysis tool should probably be able to detect them statically.
A more comprehensive, and more importantly, a static analysis tool, is Frama-C. See the paper entitled “Test-driving static analysis tools in search of C code vulnerabilities“. The paper describes a need for the very test suite that you ask for. It also links to a test suite and the effectiveness among common analyzers. Frama-C might have its own benchmarks, you can download the source and check.
See also:
Checkers (Short Paper)”