We’re building a plug-in for Outlook that should work on multiple Windows versions (XP, Vista, 7) and also with different Outlook versions (2003, 2007, 2010).
The testing problem I am facing right now, is that I can’t figure out a good/convenient/thorough way to test the application on multiple Windows and Outlook versions.
At the moment, I have a VirtualBox which runs many virtual machines, with different Windows versions and Outlook versions. So I would have a virtual machine with Windows 7 testing Outlook 2010, and another one with Windows 7 testing Outlook 2007, Windows Vista with Outlook 2010 and so on, going through some of the possible combinations. It kind of gets the job done, although it is cumbersome and takes a long time to test.
Some of the testing included in the application is unit testing, but this is also rather tied in with the machine I test it on (windows 7 with outlook 2010). For example, I was using ManagementObject recently, which worked fine on my system (and thus passed the unit test for that method), however, using that object threw an exception in another person’s system, which crashed the application.
I work on Visual Studio 2010 Ultimate.
The questions: Is there a more elegant way to make the testing process more streamline and more efficient? Any other testing methods you recommend? How would you deal with this problem?
Thanks! Looking forward to your replies.
I’ve worked on similar situations (in my case is to test 20 different languages of Windows). The basic idea is the same: each virtual machine maps to a specific condition that you need to test.
The cumbersome part comes from inappropriate tool usage. AFAIK, there are several ways to achieve automation (to a certain degree):