Question

I have the following test that does not fail when running an especially long fib assert.

Tests that don’t fail properly

#!/usr/env/bin python2.7

import unittest
from fib import fib
from nose.tools import timed


def test_gen(expected, actual):
    @timed(.001)
    def test_method(self):
        return self.assertEqual(expected, actual)
    return test_method

if __name__ == '__main__':
    all_cases = {
        'user': ((fib(40), 102334155), (fib(2), 1), (fib(5), 5)),
    }
    fails = {}
    for username, cases in all_cases.items():
        class FibTests(unittest.TestCase):
            pass

        for index, case in enumerate(cases):
            test_name = 'test_{0}_{1}'.format(username, index)
            test = test_gen(case[1], case[0])
            setattr(FibTests, test_name, test)

        suite = unittest.TestLoader().loadTestsFromTestCase(FibTests)
        result = unittest.TextTestRunner(verbosity=2).run(suite)
        fails[username] = len(result.failures)
    print fails

(Slow) Fib.py Implementation

def fib(x):
    if x == 0:
        return 0
    elif x == 1:
        return 1

    return fib(x - 2) + fib(x - 1)

Tests that fail properly

import unittest
from fib import fib
from nose.tools import timed


def test_gen(expected, actual):
    @timed(.001)
    def test_method(self):
        time.sleep(.2)
        return self.assertEqual(expected, actual)
    return test_method

Answer 1

You are timing the wrong thing, and never actually calling your test method. You are also going to an awful lot of effort to dynamically create and add your cases to your class that does nothing but act as a container for tests when nose supports generator test cases, which would be much easier to read and follow than what you have here. Also, is this a test file or a piece of product code? If it’s a test file, then having all of that code in if __name__ == '__main__' is kind of odd, and if it is a product code file, then having the test_gen function and the unittest and nose import statements in the uncoditionally run part doesn’t make much sense. I’d recommend doing it the following way, and not trying to make the test script self-runnable; just launch it with nose.

from fib import fib
from nose.tools import timed

fib = timed(.001)(fib)

def execute(username, fib_arg, expected_output):
    result = fib(fib_arg)
    assert result == expected_output, ('%s fib(%d) got %d, expected %d'
                                       % (username, fib_arg, result, expected_output))

def test_fib():
    for name, datasets in (('user', ((40, 102334155), (2, 1), (5, 5))),):
        for arg, expected in datasets:
            yield execute, name, arg, expected