Question

I have a JSON parser library (ijson) with a test suite using unittest. The library actually has several parsing implementations — “backends” — in the form of a modules with the identical API. I want to automatically run the test suite several times for each available backend. My goals are:

• I want to keep all tests in one place as they are backend agnostic.
• I want the name of the currently used backend to be visible in some fashion when a test fails.
• I want to be able to run a single TestCase or a single test, as unittest normally allows.

So what’s the best way to organize the test suite for this? Write a custom test runner? Let TestCases load backends themselves? Imperatively generate separate TestCase classes for each backend?

By the way, I’m not married to unittest library in particular and I’m open to try another one if it solves the problem. But unittest is preferable since I already have the test code in place.

Answer 1

One common way is to group all your tests together in one class with an abstract method that creates an instance of the backend (if you need to create multiple instances in a test), or expects setUp to create an instance of the backend.

You can then create subclasses that create the different backends as needed.

If you are using a test loader that automatically detects TestCase subclasses, you’ll probably need to make one change: don’t make the common base class a subclass of TestCase: instead treat it as a mixin, and make the backend classes subclass from both TestCase and the mixin.

For example:

class BackendTests:
    def make_backend(self):
        raise NotImplementedError

    def test_one(self):
        backend = self.make_backend()
        # perform a test on the backend

class FooBackendTests(unittest.TestCase, BackendTests):
    def make_backend(self):
        # Create an instance of the "foo" backend:
        return foo_backend

class BarBackendTests(unittest.TestCase, BackendTests):
    def make_backend(self):
        # Create an instance of the "bar" backend:
        return bar_backend

When building a test suite from the above, you will have independent test cases FooBackendTests.test_one and BarBackendTests.test_one that test the same feature on the two backends.