Question

I’ve been reading a bit about acceptance testing and I’d like to add some to our web application. We been experiencing a number of regressions recently and I think that this would help us.

We’re using the MVC4 framework to deliver views, but also using knockout.js and the webapi to drive the generation of the html through json data. It’s quite a javascript heavy application.

I’ve been reading a bit about SpecFlow and I think it looks good. Our testers could write their test cases in the gerkin syntax then leave the programmers to implement them or something along those lines.

I have a few questions about this and maybe my understanding is a bit limited here. Maybe this is a bad question in general but any advice on this would be muchly appreciated.

1. Is acceptance testing end-to-end testing of the live product? I’ve read some yes and some no on this.
2. Is it possible to write tests with Specflow that manipulate the browser?
3. Should I be testing end-to-end or should I just unit test the c#, the javascript and maybe throw in some integration tests on the webapi routes?
4. Should I be using something like Ranorex or Selenium for end-to-end?

Answer 1

SpecFlow is a framework for allowing developers and business folk to collaborate on defining the business functionality of the system in the form of executable specifications that can be run repeatably and consistently and serve as a ‘living documentation’ system defining how the system is supposed to behave at any given point. To do this effectively the specifications need to be written from the outside in, i.e. hitting the entry points to your application, which for web based applications would either be a website URL or some kind of service endpoint, and validating the business requirements for that end point. To answer your questions:

Is acceptance testing end-to-end testing of the live product? I’ve read some yes and some no on this.
Should I be testing end-to-end or should I just unit test the c#, the javascript and maybe throw in some integration tests on the webapi routes?

Acceptance testing is a large part of it but shouldn’t be done in isolation. You may well require integration tests as well that actually hit your data store if you’re using one or verify that external systems are communicating as expected. You should certainly be unit testing your individual components and mocking/stubbing out dependencies as appropriate, but this won’t tell you if the different layers of the system are functioning interdependently as a whole. This is where SpecFlow really shines. I wouldn’t necessarily advocate running acceptance tests on your live site, but setting up a mirror of live specifically for validating the acceptance tests would be a very worthwhile endeavour, ideally as part of your CI build process.

Is it possible to write tests with Specflow that manipulate the browser?
Should I be using something like Ranorex or Selenium for end-to-end?

Absolutely it is possible. You can use Selenium WebDriver or something like Watin (a .NET wrapper around Selenium). Watin only worked in IE the last time I looked at it. It was working with Firefox 3.6 but a change in the FF core when they released 5 stopped it working. A pity as it was a really nice framework. SpecFlow provides hooks that you can use to spin up and tear down the browser during each test run. Here’s a quick example using Selenium WebDriver:

[Binding]
public class FeatureHooks
{
    [BeforeFeature]
    public static void StartWebDriver()
    {
        Runner.InitialiseHost(HostTypeOption.Firefox);
    }

    [AfterFeature]
    public static void StopWebDriver()
    {
        Runner.StopHost();
    }

}

The Runner is a static class I use as part of a page framework I’m developing to help with the acceptance test automation. Here’s the host initialisation part:

public static class Runner
{
    internal static IWebDriver Host { get; private set; }

    public static void InitialiseHost(HostTypeOption hostType, int windowWidth = 1024, int windowHeight = 768)
    {
        switch (hostType)
        {
            case HostTypeOption.Firefox:
                Host = new FirefoxDriver();
                break;
            case HostTypeOption.InternetExplorer:
                Host = new InternetExplorerDriver();
                break;
            case HostTypeOption.Chrome:
                Host = new ChromeDriver();
                break;
            case HostTypeOption.Safari:
                Host = new SafariDriver();
                break;
            default:
                throw new ArgumentOutOfRangeException("hostType");
        }
        Host.Manage().Window.Size = new System.Drawing.Size(windowWidth, windowHeight);
        Host.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(3));
    }

    public static void StopHost()
    {
        Host.Quit();
        Host = null;
    }
}

If you want to use IE or Chrome, you’ll need to download a specific driver and add it to the bin directory of your Acceptance Test project but Firefox works as is.