Question

What is the best way to measure coverage for your Acceptance Tests??

How do you define how much your acceptance tests cover and when they are enough??

Answer 1

When doing acceptance testing, I’d rather look at the functional coverage i.e. how much features (or user stories or use cases) of a given application are tested. And to me, for each feature (or user story or use case), all scenari or possible paths should be tested. In other words, features should be the starting point to write a test plan and the goal should be to cover features, not code. This is a slight difference but an important one: measuring acceptance testing with code-coverage is developer oriented, measuring functional coverage is end-user oriented.

Regarding measurement, Ron Jeffries, one of the founding fathers of Agile, wrote an excellent summary of what he considers a truly valuable metric, RTF or Running Tested Features. You can view the article here. I’m just quoting a small part below:

What is the Point of the Project?

I’m just guessing, but I think the
point of most software development
projects is software that works, and
that has the most features possible
per dollar of investment. I call that
notion Running Tested [Features], and
in fact it can be measured, to a
degree.

Imagine the following definition of
RTF:

1. The desired software is broken down into named features
   (requirements, stories) which are part
   of what it means to deliver the
   desired system.
2. For each named feature, there are one or more automated acceptance
   tests which, when they work, will show
   that the feature in question is
   implemented.
3. The RTF metric shows, at every moment in the project, how many
   features are passing all their
   acceptance tests.

How many customer-defined features are
known, through independently-defined
testing, to be working? Now there’s a
metric I could live with.

To report progress, I especially like cumulative flow diagrams. They clearly show when something is going wrong, e.g. (here a burnup style):

_{(source: xprogramming.com)}

For a real life example, have a look at this post or this one (burndown style chart).