What are some situations where unit testing and TDD and the like are more trouble than they’re worth?
Some things I’ve come up with are:
- When generating test data is tricky: Sometimes, being able to come up with valid, non trivial test data is a challenge in itself.
- When the only practical way of verifying correctness of the code is to run it.
- When you’re testing visual elements of the design.
What are some other cases?
I think Michael summed it up quite nicely: ‘things that cannot really be formally specified’. Turns out there are lots of things that cannot be formally specified. Usability is one example (although once you decided which behavior is usable, you can and should of course test that behavior!). Somewhat paradoxically, lots of numbercrunching tasks cannot be formally specified: Take for example a weather forecast. The goal is of course to predict tomorrow’s weather, but that’s not a formal specification. So you can either test if the algorithms you use do what they should do (calculating averages, inverting matrices, stuff that can be formally specified), but then your weather forecast program could pass all the tests and still be wrong 90% of the time. Or you could use lots of historical data to test if the algorithm yields good predictions, but this is dangerous, because it can easily lead to an algorithm that’s only accurate for the historical data you used, not in general. And it would probably mean that your unit tests take hours or days to run. Even worse, your algorithm might have parameters that have to be ‘tweaked’ e.g. for the measurement instruments used, and the optimal parameters might not be the same ones for every algorithm, so the unit-tests would need manual interaction for finding good parameters. Possible in theory, but probably not very useful. I guess the same arguments would apply to OCR, ICR, many signal processing task, face recognition (and many other image processing tasks), typical photoshop tools like ‘red eye removal’, or search engine ranking algorithms (just to name a few examples).