Question

I have written a test with WatiN and when a step through the code with F10, the test succeeds, but when I execute the ‘Run Test’ command from the context menu, the test fails.

Here’s my test:

[TestMethod]
[STAThread]
public void Should_show_captcha_after_three_invalid_login_attempts_with_invalid_username()
{
    // Given
    int numberOfLoginAttempts = 3;

    // When
    for (int loginAttempt = 1; loginAttempt <= numberOfLoginAttempts; loginAttempt++)
    {
        EnterUsername(LoginSettings.ValidUserName);
        EnterPassword(loginAttempt.ToString());

        ClickLoginButton();

        // Check we are still on the loginpage
        Assert.IsTrue(_browser.Title.Contains("Inloggen"));
    }

    bool isCaptchaVisible = _browser.Page<LoginPage>().Captcha.Exists;

    // Then
    Assert.IsTrue(isCaptchaVisible);

    // Make sure to clear the login attempts for next test cases
    RemoveLoginAttempts();
}

FYI: In the DB we keep track of the loginAttempts based on the username. When the number of loginattempts is > 2, the captcha is shown. The problem I encounter is that the counter in the DB stays 1. When I manually step through the test, the counter is increased.

How is this possible?

Answer 1

Well you are right it’s got to be timing. However part of the problem is this is not a unit test, and the other is there’s a great deal of asynchronous stuff going on that you are assuming will have completed before you execute another line of test code which has been written on the assumption that it has.

For instance the count in your loop for login attempts is definitely not the one in the db. All things being equal it should match up, but..

So to me you should have a test of the login function.
That logins up the colunt in the db if unsucessful, and reset that count if it succeeds.
Then an other test to see that when login attempts in the db has gone over the limit, login response detects that and shows the the correct response.

If you want to join all this up for an end to end / whitebox test. Then an automation test of some description should be used.

I suspect that windows, the browser, your webserver or even your dbms didn’t get time to finish processing the first login attempt, before you’d queued up another two, and then done the test. Whereas in debug mode inbetwen you stepping through, they have plenty of time.