Question

Background

I’m trying to figure out how to run a single unittest against multiple input values and then display the failure. This is a trivial demo of the sort of thing I have in mind:

from time import time
import unittest

def demo():
    while True:
        count = 0
        for i in xrange(10):
            count += 1
            yield int(time() * 1000) + count
        count = 0

class TestDemo(unittest.TestCase):
    def setUp(self):
        self.gen = demo()
        self.prev = next(self.gen)

    def test_always_bigger(self):
        for cycle in xrange(1000):
            curr = next(self.gen)
            self.assertGreater(curr, self.prev)
            self.prev = curr

if __name__ == '__main__':
    unittest.main()

The most similar questions I’ve found been answered with some sort of dynamic test_<something> method creation (ex: 1, 2, 3) or nose generators (ex: 1, 2). I’m looking to run 1000s of iterations based on unpredictable input while sticking to the standard library so neither solution is a great fit. Simple looping (as shown above) works well with two limitations: for complex inputs there is an inadequate record of what caused the test to fail (my real problem); and, a single failure within the test_<...> method fails the whole test.

Question

I can live with the early failure, but how can I get at the inputs causing a failure without creating thousands of lines of output on success?

Non-answer

I tried the assert... method msg kwarg but it really only works well for the sort of trivial cases the library already handles well. In the above example unittest knows to show the two longs that caused the assertion failure. I could annotate that and provide a lot of insight into the failure with `self.assertGreater(curr, self.prev, msg=”cycle %s” % cycle), but showing deeply nested dicts on an assertDictEquals is a mess.

Desirable Solution

This answer shows an interesting use of the logging module but would generate thousands of lines of output for each successful test. Is it possible to detect failure from within the test method? Something like:

    def test_always_bigger(self):
        for cycle in xrange(1000):
            curr = next(self.gen)
            fail = self.assertGreater(curr, self.prev)
            if fail:
                log.debug('...')
            self.prev = curr

Answer 1

def test_always_bigger(self):
    for cycle in xrange(1000):
        curr = next(self.gen)
        try:
            self.assertGreater(curr, self.prev)
        except AssertionError: # raised by TestCase.fail, called by all asserts
            log.debug('...')
            raise
        self.prev = curr

This implements the semantics of logging in case of failure, then continuing with the failure. If you want to finish all tests, I would do the following:

def test_always_bigger(self):
    ex = None
    for cycle in xrange(1000):
        curr = next(self.gen)
        try:
            self.assertGreater(curr, self.prev)
        except AssertionError, ae:
            ex = ae # just remember it
            log.debug('...')
        self.prev = curr
    if ex:
        raise ex

Obviously, this would only raise the first AssertionError, but it would run until completion and would log all failures independently. Since this function represents only one test for the framework, you can’t really produce multiple failures out of it.

If you need access to all the exceptions for some reason, you may be able to get away with the following (not tested thoroughly with unittest)

def test_always_bigger(self):
    exes = []
    for cycle in xrange(1000):
        curr = next(self.gen)
        try:
            self.assertGreater(curr, self.prev)
        except AssertionError, ae:
            exes.append(ae)
            log.debug('...')
        self.prev = curr
    if exes:
        self.fail(exes)
        # if that doesn't work, try: raise AssertionError(exes)