Question

I need to setup a testing infrastructure to test combinations of cookies enabled/disabled, Sliverlight and Flash plug-ins enabled/disabled, and javascript enabled/disabled. For starters, I need to test on IE6, IE7, IE8, IE9, Chrome, FF 3.6, FF 4.0, Safari 5 and Opera 10 on their most likely OS.
Longer term, I get as close as possible to testing on every “relevant” browser including mobile devices. I’ll also want to include testing IE, for example, on XP, Vista and Win 7.

Even if I pick a single OS per browser, I have more than 100 host setups. What tools or strategies can bring sanity to this process?

(My current thinking suggests that screenshots alone will not be sufficient. It could be that I can develop a basic smoke test which gives me some indication of a problem and is verifiable through screenshot alone.)

Answer 1

Have you looked at Selenium, specifically Selenium Grid? It’s not going to cover every single one of your test setups, but it’ll get you a lot of the way there.

Selenium Grid transparently distribute [sic] your tests on multiple machines so that you can run your tests in parallel, cutting down the time required for running in-browser test suites. This will dramatically speeds up in-browser web testing, giving you quick and accurate feedback you can rely on to improve your web application.

See also:

• Which is the most comprehensive browser compatibility testing tool?
• How to automate testing of a browser-based app?
• https://stackoverflow.com/questions/2565505/whats-the-best-way-to-do-cross-browser-testing

All that said:

It really might not be worth testing every single browser/version/OS combination. Part of this is because, at least on desktop OSes, most variations will show up across browsers (and to a lesser extent, browser version), and have much less to do with the OS version.

Chances are that the users of your site will not have equally distributed browsers/versions/OSes. It’s basically a case of the 80-20 rule: the majority of your users will be using the minority of browser/version/OS combinations. So, at least in the short term, you get much more for your effort if you focus on ensuring that the site works on only the most popular OSes and browsers.

I’d also expect that your site, like many mobile-friendly sites, has a separate mobile view (reduced JavaScript, less content per page, simpler CSS, etc.). In that case, you’d only need to test the mobile version on mobile browsers and OSes, and the full version of the site on desktop browsers/OSes.