Question

I am looking for good academic references on how to benchmark programs. There seems to be a lot of lore in benchmarking, but I haven’t seen many references that explain what a good benchmark is, what a bad one is, and how to write one.

Thanks.

Answer 1

Academically speaking, a relevant article is “Statistically rigorous Java performance evaluation” from OOPSLA 2007 (which you can find from Google Scholar); while focused on Java, it contains general lessons on benchmarking, and the content about Java generalizes nicely to most languages running on some virtual machine and simply using garbage collection. Finally, they summarize the statistics knowledge needed for analyzing the results.

Additionally, here is a framework from Google:

http://code.google.com/p/caliper/

And here their Wiki discusses some criteria for a good benchmark:
http://code.google.com/p/caliper/wiki/JavaMicrobenchmarkReviewCriteria