Question

How can I run the same test against a lot of different data?

I want to be reported of all failures.

For example:

def isEven(number):
    return True # Quite buggy implementation

data = [
    (2, True),
    (3, False),
    (4, True),
    (5, False),
]

class MyTest:
   def evenTest(self, num, expected):
       self.assertEquals(expected, isEven(num))

I have found solution which raises error on first failure only:
PHPUnit style dataProvider in Python unit test

How can I run a test to be reported of all failures?

Answer 1

You should be using py.test. I think the unittest module was blindly copied from JUnit. Anyway, you can hack your way like this:

import unittest

data = [
    (2, True),
    (3, False),
    (4, True),
    (5, False)]

# This should be imported from a separate module.
def isEven(number):
    return True # Quite buggy implementation

def create_test_func(num, expected):
    def _test_func(self):
        self.assertEqual(expected, isEven(num))
    return _test_func

class TestIsEven(unittest.TestCase):

    pass

# pyunit isn't Pythonic enough. Use py.test instead
# till then we rely on such hackery
import new
for i, (num, expected) in enumerate(data):
    setattr(TestIsEven, 'test_data_%d'%i, create_test_func(num, expected))

if __name__ == "__main__":
    unittest.main()

And the output is:

.F.F
======================================================================
FAIL: test_data_1 (__main__.TestIsEven)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "untitled-1.py", line 15, in _test_func
    self.assertEqual(expected, isEven(num))
AssertionError: False != True

======================================================================
FAIL: test_data_3 (__main__.TestIsEven)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "untitled-1.py", line 15, in _test_func
    self.assertEqual(expected, isEven(num))
AssertionError: False != True

----------------------------------------------------------------------
Ran 4 tests in 0.000s

FAILED (failures=2)

Using this approach, you can add more niceties like printing debugging information on failure, etc.