Recently I deployed some software onto a client’s PC. The software built and ran correctly on my dev machine and passed all unit and code quality checks on the build server but it crashed explosively when run on the client’s PC.
I narrowed the problem down to an external .NET library which referenced a native library that was only included in the .NET Framework 3.5, and the client’s PC only had .NET Framework 4.0 installed. (The culprit was Sql Server CE 3.5).
What would be an effective way to detect this kind of issue proactively? I could use an automated UI testing framework running on a clean environment which could be remotely invoked by the build server. This could also be a segue into a set of GUI tests in addition to the goal of answering “Does it run on this environment?” but I’m concerned the two objectives should have clearer separation rather than being lumped together into one set of automated tests. Is there a better way? Also, is there a name for this kind of platform compatibility test?
I found it was overkill to automate the process of testing on a clean environment since it only has to be done once per public release. I settled for a Virtual PC image of my deployment environment with an “Undo disk” which can be reverted to a clean state after each test run.