Question

Over time I found the need to override several stdlib methods from Python in order to overcome limitation or to add some missing functionality.

In all cases I added a wrapper function and replaced the original method from the module with my wrapper (the wrapper was calling the original method).

Why I did this? Just to be sure that all the calls to the method are using my new versions, even if these are called from other third-party modules.

I know that monkeypatching can be a bad thing but my question is if this is useful if you use it with care? Meaning that:

• you still call the original methods, assuring that you are not missing anything when the original module is updated
• you are not changing the original “meaning” of the methods

Examples:

• add coloring support to python logging module.
• make open() be able to recognize Unicode BOM masks when using text mode
• adding logging support to os.system() or subprocess.Popen() – letting you output to console or/and redirect to another file.
• implementing methods that are missing on your platform like os.chown() or os.lchown() that are missing on Windows.

Doing things like these appear to me as decent overrides but I would like to see how others are seeing them and specially what should be considered as an acceptable monkeypatch and what not.

Answer 1

Monkeypatching can be “the least of evils”, sometimes — mostly, when you need to test code which uses a subsystem that is not well designed for testability (doesn’t support dependency injection &c). In those cases you will be monkeypatching (very temporarily, fortunately) in your test harness, and almost invariably monkeypatching with mocks or fakes for the purpose of isolating tests (i.e., making them unit tests, rather than integration tests).

This “bad but could be worse” use case does not appear to apply to your examples — they can all be better architected by editing the application level code to call your appropriate wrapper functions (say myos.chown rather than the bare os.chown, for example) and putting your wrapper functions in your own intermediate modules (such as myown) that stand between the application level code and the standard library (or third-party extensions that you are thus wrapping — there’s nothing special about the standard library in this respect).

One problematic situation might arise when the “application level code” isn’t really under your control — it’s a third party subsystem that you’d rather not modify. Nevertheless, I have found that in such situations modifying the third party subsystem to call wrappers (rather than the standard library functions directly) is way more productive in the long run — then of course you submit the change to the maintainers of the third party subsystem in question, they roll your change into their subsystem’s next release, and life gets better for everybody (you included, since once your changes are accepted they’ll get routinely maintained and tested by others!-).

(As a side note, such wrappers may also be worth submitting as diffs to the standard library, but that is a different case since the standard library evolves very very slowly and cautiously, and in particular on the Python 2 line will never evolve any longer, since 2.7 is the last of that line and it’s feature-frozen).

Of course, all of this presupposes an open-source culture. If for some mysterious reasons you’re using a closed-source third party subsystem, therefore one which you cannot possibly maintain, then you are in another situation where monkey patching may be the lesser evil (but that’s just because the evil of losing strategic control of your development by trusting in code you can’t possibly maintain is such a bigger evil in itself;-). I’ve never found myself in this situation with a third-party package that was both closed-source and itself written in Python (if the latter condition doesn’t hold your monkeypatches would do you no good;-).

Note that here the working definition of “closed-source” is really very strict: for example, even Microsoft 12+ years ago distributed sources of libraries such as MFC with Visual C++ (as their product was then called) — closed-source because you couldn’t redistribute their sources, but still, you DID have sources at hand, so when you met some terrible limitation or bug you COULD fix it (and submit the change to them for a future release, as well as publishing your change as a diff as long as it included absolutely none of their copyrighted code — not trivial, but feasible).

Monkeypatching well beyond the strict confines within which such an approach is “the least of evil” is a frequent mistake of users of dynamic languages — be careful not to fall into that trap yourself!