Question

A while ago I had to upgrade some servers from Python 2.4 to Python 2.5. I found that .pyc files created under Python 2.4 would crash when Python 2.5 tried to run them.

Will this happen again when I upgrade from 2.5 to 2.6?

EDIT: Here is a bit more detail

I have a fileserver that contains the python code. This is accessed by both Ubuntu and Windows servers to run the python code. When they run the code they produce .pyc files on the fileserver.

I found that when I upgraded one of the server machines from Python 2.4 to 2.5 I had problems with .pyc files. I’m now not sure whether it was a machine running 2.5 that tried to run 2.4 bytecode or whether it was a 2.4 machine trying to run 2.5 bytecode, but if I deleted the bytecode all went well until the next bytecode clash.

I upgraded all of the machines to 2.5 and the problem went away.

Answer 1

In general, .pyc files are specific to one Python version (although portable across different machine architectures, as long as they’re running the same version); the files carry the information about the relevant Python version in their headers — so, if you leave the corresponding .py files next to the .pyc ones, the .pyc will be rebuilt every time a different Python version is used to import those modules. “Trying to run” wrong-version .pyc files is something I never heard about. What architectures were involved? Were the .py files around as they should be?

Edit: as the OP clarified that the crashes came when he was running both Python 2.4 and Python 2.5 programs on the same .py files (from two different servers, sharing a network drive), the explanation of the crashes becomes easy. The .py files were all the time being recompiled — by the 2.4 Python when the 2.5 had been the one running them most recently, and vice versa — and therefore the .pyc files were frantically busy getting rewritten all the time. Proper file locking on network drives (especially but not exclusively across different operating systems) is notoriously hard to achieve. So the following must have happened (the roles could be switched): the 2.4 server had just determined that a .pyc file was fine for it and started reading it; before it could finish reading, the 2.5 server (having previously determined that the module needed to be recompiled) wrote over it; so the 2.4 server ended up with a memory buffer that had (say) the first 4K bytes from the 2.4 version and the next 4K bytes from the 2.5 version. When it then used that mangled buffer, unsurprisingly… crash!!!

This can be a real problem if you ever find yourself continuously trying to run a single set of .py files from two or more different versions of Python (even on the same server, without the added complications of network drives). The “proper” solution would be something like virtualenv. The (simple, but dirty-ish) hack we adopted at work (many years ago, but it’s still in production…!) is to patch each version of Python to produce and use a different extension for its compiled bytecode files: .pyc (or .pyo) for Python 1.5.2 (which was the most stable “system” version back when we started doing this kludge to newer versions), .pyc-2.0 for 2.0, .pyc-2.2 for 2.2, and so forth (or equivalent .pyo-X.Y of course). I hear this is soon going away at long last (thanks Thomas!-), but it did tide us semi-decently over this ticklish problem for many, many years.

A much simpler solution is to keep a single version of Python around, if that’s feasible for your system; if your system has any complications that make it unfeasible to have a single Python version (as ours did, and does), then these days I’d heartily recommend virtualenv, which I’ve already mentioned.

---

With the adoption of PEP 3147 in Python 3.2, pyc files for different Python versions are distinguished automatically by filename. This should solve most problems with different Python versions overwriting each other’s pyc files.