Question

I maintain an installable Django app that includes a regular test suite.

Naturally enough when project authors run manage.py test for their site, the tests for both their own apps and also any third party installed apps such as mine will all run.

The problem that I’m seeing is that in several different cases, the user’s particular settings.py will contain configurations that cause my app’s tests to fail.

A couple of examples:

• Some of the tests need to check for returned error messages. These error messages use the internationalization framework, so if the site language is not english then these tests fail.
• Some of the tests need to check for particular template output. If the site is using customized templates (which the app supports) then the tests will end up using their customized templates in preference to the defaults, and again the tests will fail.

I want to try to figure out a sensible approach to isolating the environment that my tests get run with in order to avoid this.

My plan at the moment is to have all my TestCase classes extend a base TestCase, which overrides the settings, and any other environment setup I may need to take care of.

My questions are:

• Is this the best approach to app-level test-environment isolation? Is there an alternative I’ve missed?
• It looks like I can only override a setting at a time, when ideally I’d probably like a completely clean configuration. Is there be a way to do this, and if not which are the main settings I need to make sure are set in order to have a basic clean setup?
• I believe I’m correct in saying that overriding some settings such as INSTALLED_APPS may not actually affect the environment in the expected way due to implementation details, and global state issues. Is this correct? Which settings do I need to be aware of, and what globally cached environment information may not be affected as expected?
• What other environment state other than settings might I need to ensure is clean?

More generally, I’d also be interested in any context regarding how much of an issue this is for other third party installable apps, or if there are any plans to further address any of this in core. I’ve seen conversation on IRC regarding similar issues with eg. some of Django’s contrib apps running under unexpected settings configurations. I seem to also remember running into similar cases with both third party apps and django contrib apps a few times, so it feels like I’m not alone in facing these kind of problems, but it’s not clear if there’s a consensus on if this is something that needs more work or if the status quo is good enough.

Note that:

• These are integration-level tests, so I want to address these environment issues at the global level.
• I need to support Django 1.3, but can put in some compatibility wrappers so long as I’m not re-implementing massive amounts of Django code.
• Obviously enough, since this is an installable app, I can’t just specify my own DJANGO_SETTINGS_MODULE to be used for the tests.

Answer 1

A nice approach to isolation I’ve seen used by Jezdez is to have a submodule called my_app.tests which contains all the test code (example). This means that those tests are NOT run by default when someone installs your app, so they don’t get random phantom test failures, but if they want to check that they haven’t inadvertently broken something then it’s as simple as adding myapp.tests to INSTALLED_APPS to get it to run.

Within the tests, you can do your best to ensure that the correct environment exists using override_settings (if this isn’t in 1.4 then there’s not that much code to it). Personally my feeling is that with integration type tests perhaps it doesn’t matter if they fail. If you like, you can include a clean settings file (compressor.test_settings), which for a major project may be more appropriate.

An alternative is that you separate your tests out a bit – there are two separate bodies of tests for contrib.admin, those at django.contrib.admin.tests, and those at tests.regression_tests.contrib.admin (or some path like that). The ones to check public apis and core functionality (should) reside in the first, and anything likely to get broken by someone else’s (reasonable) configuration resides in the second.

IMHO, the whole running external apps tests is totally broken. It certainly shouldn’t happen by default (and there are discussions to that effect) and it shouldn’t even be a thing – if someone’s external app test suite is broken by my monkey patching (or whatever) I don’t actually care – and I definitely don’t want it to break the build of my site. That said, the above approaches allow those who disagree to run them fairly easily. Jezdez probably has as many major pluggable apps as anyone else, and even if there are some subtle issues with his approach at least there is consistency of behaviour.