This question isn’t about unit-testing. And it is for a desktop product.
This is about testing of the gui and testing that the right stuff is input in the right text box at the right time.
A company I used to work at used WinRunner (different department so I don’t know much more that that), but that has now been shutdown by HP but they don’t seem that bothered whether you stay with HP or go elsewhere. You can’t read about the product until you’ve signed up which is annoying.
The tool has to work with MFC (non-negotiable) and the ideal tool will also…
- be automated.
- be scriptable.
- work with different screen resolutions automatically.
- be able to ‘spy’ on individual static text boxes, etc.
- intuitive enough so non-programmers can create the scripts.
- have reporting tools, including email of individual users.
What do other SO users do for automated GUI testing?
We use the SAFS framework for Rational Robot (RRAFS). There are also SAFS implementations for WinRunner (WRAFS) and it looks like they have a new ‘Image-Based Testing’ implementation, which I’m not familiar with.
This framework does a nice job of seperating the UI implementation from the test scripts. I’ve tested four releases of a web application developed by two different teams (one team using classic ASP, one using ASP.NET) and I only had to change the application map of my UI objects, the tests themselves didn’t need to change.
That said, the language of the framework is cumbersome and takes getting used to. It’s not very robust, in terms of language constructs, but with some effort you can do anything you need to. It’s sort of like ‘programming’ in Windows Batch language, but for tests 😉
To address your individual requirements above:
1) The tool has to work with MFC (non-negotiable). The SAFS framework uses a 3rd party ‘record-playback’ tool to drive the tests, like Rational Robot or Mercury WinRunner. If that tool can interact with MFC apps, then the framework can. I don’t know how the ‘Image-Based Testing’ implementation drives the tests, but I’d guess it can also work with MFC.
2) be automated. The SAFS framework integrates with the STAF framework, which will allow you to automate the execution of your tests. I have a proof-of-concept test that uses STAF to start a VM image from a pool of images, install the application under test, run the RRAFS test, and put the results on a web server for others to get at.
3) be scriptable. Yes, but as mentioned, it’s not the most robust programming language. I wrote an Excel add-in that our testers use to write their tests that simplifies things a little bit.
4) work with different screen resolutions automatically. Yes, since it’s looking ‘under the covers’ at the UI objects and not the screen. Except for maybe the ‘Image-based Testing’ option…
5) be able to ‘spy’ on individual static text boxes, etc. Yes, you can wait for a UI object to appear, disapper, to have a value, for a value to be changed, etc.
6) intuitive enough so non-programmers can create the scripts. With some training. We’ve had limited success. Some QA folks can write the tests, some struggle.
7) have reporting tools, including email of individual users. Yes, using the STAF framework you can post results to a web server, send out emails, etc.