Question

How can I mark a test as an expected failure in JUnit 4?

In this case I want to continue to run this test until something is patched upstream. Ignoring the test goes a little too far, as then I might forget about it. I may be able to add an @expected annotation and catch the exception thrown by assertThat, but that also seems to lie about the expected behavior.

Here’s what my current test looks like:

@Test
public void unmarshalledDocumentHasExpectedValue() 
{
    doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));
    final ST title = doc.getTitle();
    assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}

That assert should succeed, but because of an upstream bug it doesn’t. Yet, that test is correct; it should succeed. Virtually all the alternatives that I’ve found are misleading. Right now I think @Ignore("This test should pass once fixed upstream") is my best bet, but I still have to remember to come back to it. I’d prefer that the test run.

In Python I can use the expectedFailure decorator:

class ExpectedFailureTestCase(unittest.TestCase):
    @unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0, "broken")

With Qt’s QTestLib in C++, you can use QEXPECT_FAIL:

QEXPECT_FAIL("", "Will be fixed next version", Continue);
QCOMPARE(i, 42);

In both cases above, the unit test runs which is what I’m hoping to have happen. Am I missing something in JUnit?

Answer 1

I’m assuming here that you want the test to pass if your assert fails, but if the assert succeeds, then the test should pass as well.

The easiest way to do this is to use a TestRule. TestRule gives the opportunity to execute code before and after a test method is run. Here is an example:

public class ExpectedFailureTest {
    public class ExpectedFailure implements TestRule {
        public Statement apply(Statement base, Description description) {
            return statement(base, description);
        }

        private Statement statement(final Statement base, final Description description) {
            return new Statement() {
                @Override
                public void evaluate() throws Throwable {
                    try {
                        base.evaluate();
                    } catch (Throwable e) {
                        if (description.getAnnotation(Deprecated.class) != null) {
                            // you can do whatever you like here.
                            System.err.println("test failed, but that's ok:");
                        } else {
                            throw e;
                        }
                    }
                }
            };
        }
    }

    @Rule public ExpectedFailure expectedFailure = new ExpectedFailure();

    // actually fails, but we catch the exception and make the test pass.
    @Deprecated
    @Test public void testExpectedFailure() {
        Object o = null;
        o.equals("foo");
    }

    // fails
    @Test public void testExpectedFailure2() {
        Object o = null;
        o.equals("foo");
    }
}

First, note that the first method is marked as @Deprecated. I’m using this as a marker for the method for which I want to ignore any assertion failures. You can do whatever you like to identify the methods, this is just an example.

Next, in the ExpectedFailure#apply(), when I do the base.evaluate(), I’m catching any Throwable (which includes AssertionError) and if the method is marked with the annotation @Deprecated, I ignore the error. You can perform whatever logic you like to decide whether you should ignore the error or not, based on version number, some text, etc. You can also pass a dynamically determined flag into ExpectedFailure to allow it to fail for certain version numbers:

public void unmarshalledDocumentHasExpectedValue() {
    doc = unmarshaller.unmarshal(getResourceAsStream("mydoc.xml"));

    expectedFailure.setExpectedFailure(doc.getVersionNumber() < 3000);

    final ST title = doc.getTitle();
    assertThat(doc.getTitle().toStringContent(), equalTo("Expected"));
}

For further examples, see ExternalResource, and ExpectedException

Ignoring an expected failure test rather than passing it

If you want to mark you tests as Ignored rather than Success, it becomes a bit more complex, because tests are ignored before they are executed, so you have to retrospectively mark a test as ignored, which would involve constructing your own Runner. To give you a start, see my answer to How to define JUnit method rule in a suite?. Or ask another question.