Question

My question is focused on if my setup is currently following a best practice approach regarding BDD with UI acceptance testing. I’m using WatiN with SpecFlow to build my UI acceptance tests and I’m deploying my application to AppHarbor (a cloud platform as a service for .Net applications). AppHarbor runs your unit/integration tests upon deployment and only pushes your site live if your tests pass. So I started by first writing a basic failing test as follows:

Scenario: Navigation to homepage
          When I navigate to /
          Then I should be on the application homepage

The steps associated with this test open up a browser using WatiN and verify the title attribute of the view is set to “Welcome”. I am doing a check on the environment to decide what URL to test against with the WatiN browser. For example, if in development navigate to “http://localhost:49641/” for home. Otherwise, navigate to “http://myappharborapp.com/”.

My problem is that if you are deploying this application for the first time, the page or view actually does not exist and so the test fails (because the site is not live yet). This would also fail if for instance I would later add an “About” page view and wrote a failing test first. When I push the updates the test will fail because the “About” page does not yet exist.

My question then is: Am I not following best practices with regards to how you should be setting up your UI tests? How should these tests be setup so that they pass in any environment?

Any insight is greatly appreciated!

Answer 1

In “traditional” watin tests I use custom attributes to specify what version of an application and what environment it is running in, and then the test is skipped if the critiera is missed.

(The code is open source in at http://testingstax.codeplex.com in the parkcalc sample > observer > environment monitor)

    internal static void CheckSetEnvironment()
    {
        Object[] attributes = Utility.GetCallerAttributes(typeof(ExecutionEnvironment), 3);

        CheckEnvironment(attributes);
    }

    private static void CheckEnvironment(Object[] attributes)
    {
        TestEnvironment = GetCurrentEnvironment();

        if (attributes.Length > 0 && !attributes.Contains(new ExecutionEnvironment(TestEnvironment)))
        {
            Assert.Inconclusive("This test is not designed to be executed in the '" + TestEnvironment.ToString() + "' environment.");   
        }
    }

    private static EnvironmentType GetCurrentEnvironment()
    {
        string currentEnvironment = ConfigurationManager.AppSettings["Environment"].ToLower(CultureInfo.CurrentCulture);
        EnvironmentType Environment = new EnvironmentType();

        try
        {
            Environment = (EnvironmentType)Enum.Parse(typeof(EnvironmentType), currentEnvironment, true);
        }
        catch (System.ArgumentException)
        {
            Assert.Fail(" The current environment setting in 'Environment' in the app.config is invalid.");
        }
        return Environment;
    }

The trick would then be to map a specflow action to ignore the test

“Given the test is not running in production” or something similar