Question

A fellow developer on a project I am on believes that doctests are as good as unit-tests, and that if a piece of code is doctested, it does not need to be unit-tested. I do not believe this to be the case. Can anyone provide some solid, ideally cited, examples either for or against the argument that doctests replace the need for unit-tests?

Thank you
-Daniel

EDIT: Can anyone provide a reference showing that doctesting should not replace unit-testing?

Answer 1

I (ab)used doctest in lieu of unittest, back when I started my gmpy project many years ago — you can browse its sources and see that all the functionality is thoroughly tested with doctests (the functionality’s supplied by a C-coded Python extension, and last time I instrumented it for coverage measurement I was over 95% coverage). Why did I do that? Because doctest was brand new, as was gmpy, and I was curious to see how far I could push it.

Answer: very far indeed — but definitely not worth it (the novelty wears off, but you don’t want to rewrite all your tests, which is why gmpy’s tests are still all-doctests). The extreme fragility of doctests, where even the tiniest typo fix in a message breaks the test, is a real bother when they’re being abused this way. It’s kind of like traditional integration tests based on comparing output with a “golden” (known-good) expected output: easy to write the first time around, but you’ll repent at leisure after a few years of fixing gratuitous test breakages;-).

If you find unittest‘s style onerous, there are other excellent alternatives that are still meant for use in unit tests, such as py.test and nose — doctest is really meant for a different purpose (supporting docs, not generic unit tests) and while it’s of course worth adding whatever doctests you’ve written for docs purposes to your test battery, it’s not worth the test-maintenance headaches of replacing unit tests with it.