My (rather small) company develops a popular Windows application, but one thing we’ve always struggled with is testing – it frequently is only tested by the developers on a system similar to the one they developed it on, and when an update is pushed out to customers, there is a segment of our base that experiences issues due to some weird functionality with a Windows patch, or in the case of certain paranoid antivirus applications (I’m looking at you, Comodo and Kaspersky!), they will false-positive on our app.
We do manual testing on what 70% of our users use, but it’s slow and painful, and sometimes isn’t as complete as it should be. Management keeps insisting that we need to do better, but they keep punting on the issue when it comes time to release (testing will take HOW LONG? Just push it out and we’ll issue a patch to customers who experience issues!).
I’d like to design a better system of automated testing using VMs, but could use some ideas on how to implement it, or if there’s a COTS product out there, any suggestions would be great. I’m hacking a Python script together that “runs” every feature of our product, but I’m not sure how to go about testing if we get a Windows crash (besides just checking to see if it’s still in the process list), or worse yet, if Comodo has flagged it for some stupid reason.
To best simulate the test environment, I’m trying to keep the VM as “pure” as possible and not load a lot of crap on it outside of the OS and the antivirus, and some common apps (Acrobat Reader, Firefox etc).
Any ideas would be most appreciated!
Interesting problem. One thing to avoid is using the antivirus APIs to check to see if your application triggers them. You want a real live deployment of your application, on the expected operating system, with a real live AV install monitoring it. That way you’ll trigger the heuristics monitoring as well as the simple “does this code match that checksum” that the API works with.
You haven’t told us what your application is written in, but if your test suite for your application actually exercises portions of the application, rather than testing single code paths, that may be a good start. Ideally, your integration test suite is the same test suite you use to check for problems on your deploy targets. Your integration testing should verify the input AND the output for each test in a live environment, which SHOULD catch crashes and the like. Also, don’t forget to check for things that take much longer than they should, that’s an unfortunately common failure mode. Most importantly, your test suite needs to be easy enough to write, change, and improve that it actually stays in sync with the product. Tests that don’t test everything are useless, and tests that aren’t run are even worse. If we had more information about how your program works, we could give better advice about how to automate that.
You’ll probably want a suite of VM images across your intended deploy targets, in various states of patch (and unpatch). For some applications, you’ll need a separate VM for each variant of IE, since that changes other aspects of the system. Be very careful about which combination of things you have in each VM. Don’t test more than one AV at a time. Update the AVs in your snapshots before running your tests. If you have a large enough combination software in your images, you might need to automate image creation – get a base system build, update to the latest patch level, then script the installation of AV and other application combinations.
Yes, maintaining this farm of VMs will be a pain, but if you script the deploy of your application, and have good snapshots and a plan for patching and updating the snapshots, the actual test suite itself shouldn’t take all that long to run given appropriate hardware. You’ll need to investigate the VM solutions, but I’d probably start with VMWare.