Question

We are currently setting up the evaluation criteria for a trade study we will be conducting.

One of the criterion we selected is reliability (and/or robustness – are these the same?).

How do you assess that software is reliable without being able to afford much time evaluating it?

Edit: Along the lines of the response given by KenG, to narrow the focus of the question: You can choose among 50 existing software solutions. You need to assess how reliable they are, without being able to test them (at least initially). What tangible metrics or other can you use to evaluate said reliability?

Answer 1

Reliability and robustness are two different attributes of a sytem:

Reliability

The IEEE defines it as ‘. . . the ability of a system or component to perform its required functions under stated conditions for a specified period of time.’

Robustness

is robust if it continues to operate despite abnormalities in input, calculations, etc.

So a reliable system performs its functions as it was designed to within constraints; A robust system continues to operate if the unexpected/unanticipated occurs.

If you have access to any history of the software you’re evaluating, some idea of reliability can be inferred from reported defects, number of ‘patch’ releases over time, even churn in the code base.

Does the product have automated test processes? Test coverage can be another indication of confidence.

Some projects using agile methods may not fit these criteria well – frequent releases and a lot of refactoring are expected

Check with current users of the software/product for real world information.