Question

I am using JUnit 4.10 for running test suites, and I have implemented a “retry failed test” rule following Matthew Farwell’s awesome notes in the How to Re-run failed JUnit tests immediately? post. I created a class “RetryTestRule” with the following code:

public class RetryTestRule implements TestRule {

  private final int retryCount;

  public RetryTestRule(int retryCount) {
    this.retryCount = retryCount;
  }

  @Override
  public Statement apply(Statement base, Description description) {
    return statement(base, description);
  }

  private Statement statement(final Statement base, final Description description) {
    return new Statement() {
      @Override
      public void evaluate() throws Throwable {
        Throwable caughtThrowable = null;

        // retry logic
        for (int i = 0; i < retryCount; i++) {
          try {
            base.evaluate();
            return;
          } catch (Throwable t) {
            caughtThrowable = t;
            System.err.println(description.getDisplayName() + ": run " + (i + 1) + "     failed");
          }
        }
        System.err.println(description.getDisplayName() + ": Giving up after " + retryCount
            + " failures");
        throw caughtThrowable;
      }
    };
  }
}

When using this as a rule inside a test case it works perfectly, but it seems not optimal to use the @Rule notation in every test case of a suite instead of a single notation in the Suite definition, so after checking a bit I tried the new @ClassRule notation in my Suite class:

@RunWith(Suite.class)
@SuiteClasses({
  UserRegistrationTest.class,
  WebLoginTest.class
})
public class UserSuite {    
  @ClassRule
  public static RetryTestRule retry = new RetryTestRule(2);
}

Problem is this does not work as expected: failed tests are not being retried. Does anyone have tried this and knows a solution? Help is much appreciated!

Answer 1

@ClassRules are executed once per class, not once per method. To have something executed once per method, you need to use @Rule like you are doing, or follow the answer for How to define JUnit method rule in a suite?.

To reuse your existing rule, you can add the rule to the list of rules to run, using the RunRules class as follows:

public class MyRunner extends BlockJUnit4ClassRunner {
    public MyRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    @Override
    protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
        Description description= describeChild(method);
        if (method.getAnnotation(Ignore.class) != null) {
            notifier.fireTestIgnored(description);
        } else {
            RunRules runRules = new RunRules(methodBlock(method), Arrays.asList(new TestRule[]{new RetryTestRule(3)}), description);
            runLeaf(runRules, description, notifier);
        }
    }
}

This is using the example from the above answer. You could probably combine the two answers as well for more fine grained control, creating a RetryTestRule if there were an annotation on your test for example.