Question

Recently I read some articles about some doubts about benefits of acceptance testing, because it is quite costly compared to what it brings. To form my own opinion, I would like to gather as much benefits of automated acceptance testing as possible. Can you help me?

Answer 1

In general, automated testing makes sense under the condition that the total investment in setting up and maintaining the automated test for the life of the product is lower than the total investment in doing the manual test whenever needed for the life of the product.

Therefore, automated testing becomes more attractive if it is very cheap to create the test compared to running them manually (e.g., fit testing where you essentially create spreadsheets), or if there is going to be a large enough number of executions of the test to offset the cost difference.

Unit testing works well because there is a large enough number of executions – with every minor change or refactoring you want to make sure that there is no new failure.

Acceptance testing is dicier because the likelihood of repeated execution changes significantly from project to project, and business needs affect long-term support and maintenance issues.

If a project is delivered once it passes acceptance testing and there is no further support or changes, then automated acceptance testing might not make sense unless it is very cheap to produce the automated tests.

If, on the other hand, one expects future releases (even to same customer) or a long term support contract, then the initial investment may become worth it over time.

Our company produces trading systems and algorithms for a variety of customers, and new features and products are built on top of old ones. The cost of running all our tests manually is so high, that we are working on adding automated acceptance testing for many scenarios that have previously been tested manually.

Follow up:

IMHO it is important to first have very clear and complete step-by-step test plans (A good QA person or team is critical here). In the case of acceptance testing, the customers may provide their own or QA would have to provide with their customers. I believe that a test plan should often be part of the contract.

Once you have that, it is easier to estimate how long it would take a human to run it (and you can experiment on a human to find out). It is important to pay attention to what steps require “grunt work”, and what steps require real human input or discretion.

This is an important distinction. For example, suppose you are building a drawing program. if you tell a human to draw lines in specific coordinates, this can be automated easily. However, if your step says: “Verify that the shape is a flower”, a human can do it easily but automating it is almost impossible. Many test cases can be semi-automated. You leave “hooks” for user input and have a tester focus on these. A button starts a series of operations, and the user is presented with “did the correct output come out?”.

My experience is that the cost of automation is fairly intuitive. However, there is often a need to modify the existing program to make it more testable. For example, in the above example, if you have an API to allow you to add lines to the canvas, things are great. If you have to write a mouse robot that translates logical coordinates to screen and all that, it is much longer.