Question

Although TDD/BDD is ideal and as time is limited, you cannot always have 100% test coverage and always write test before implementing features, so, what is your approach to have a well tested project under a tight schedule?

My initial thinking is:

1. Acceptance tests using RSpec
2. Development
3. Back to 1

After several iterations of Acceptance tests / development cycles then I will add

1. Integration tests

Finally ship to QA for manual testing, if a bug was found, then add

1. Unit tests

Do you think the above work flow reasonable?

Answer 1

Disclaimer: i’m test-infected

It’s a sliding scale between confidence on one end and short term delivery speed on the other. You can’t have both. You can ship faster by sliding to the right. The optimal position depends on your context (team skill, project uncertainity/risks, org culture, schedule, etc.)

As for the process outline above, I might be reading it wrong. It seems you’re slicing out unit testing for speed. i.e. Do ATDD without TDD.

If that’s where you are heading, I see the following risks

• Acceptance tests are chunky: They will tell you that something failed – but not help with defect isolation. The same acceptance test may fail for myriad reasons/ Multiple system tests may fail for the same bug. You may lose a lot of time tracking down the cause. A focussed unit test should instantly tell you precisely where the problem is. Over time the time spent debugging acceptance test failures may overshoot the time saved by not writing focussed tests
• Acceptance tests are slow : As compared to unit tests, you can run only a handful (optimistic) of chunky tests in 5 mins.. where you could run thousands of micro tests in the same time. The reason this is helpful is that unit tests would give you faster feedback that is closer to the code change that broke something. Since they take time to run, the regular tendency would be to run acceptance tests the least number of times. More time spent identifying the exact code change that broke the app’s back.
• Acceptance tests are not-thorough : cannot cover every tiny edge case since it will lead to infeasible test-suite execution times. Bugs can breed in those untested spots.

Finally IMHO acceptance / system tests should be the last level of defense ; not the first. I’ve seen teams struggling with code quality issues where the team isn’t sincere about code quality and relies on the system tests to catch everything; as said above, this is fooling yourself. TDD has a more direct impact on code quality.. ATDD would just tell you that something is broken/it is ‘not acceptable’ to the customer. That said, Depending on the right mix of discipline+experience in your team, it might work in the short term.. just that I wouldn’t bet on it.

If you want to slim down the time taken for unit-testing, sit down with the team and define critical/focus areas/modules. Get those done with TDD.