Question

This started biting our ass on our production server really hard. We saw this occasionally (for 1 request per week). Back then we found out it is because of mod_wsgi doing some funky stuff in some configs. As we could not track the reason for the bug, we decided that it did not require instant attention.

However today, on 1 of our production servers this really occurred for 10 % of all server requests; that is 10 % of all server requests failed with this very same error:

mod_wsgi (pid=1718): Target WSGI script '/installation/dir/our-program/prod-dispatch.wsgi' cannot be loaded as Python module.
mod_wsgi (pid=1718): Exception occurred processing WSGI script '/installation/dir/our-program/prod-dispatch.wsgi'.
Traceback (most recent call last):
  File "/installation/dir/our-program/prod-dispatch.wsgi", line 7, in <module>
    from pyramid.paster import get_app
  File "/installation/dir/venv/local/lib/python2.7/site-packages/pyramid-1.3a6-py2.7.egg/pyramid/paster.py", line 12, in <module>
    from pyramid.scripting import prepare
  File "/installation/dir/venv/local/lib/python2.7/site-packages/pyramid-1.3a6-py2.7.egg/pyramid/scripting.py", line 1, in <module>
    from pyramid.config import global_registries
  File "/installation/dir/venv/local/lib/python2.7/site-packages/pyramid-1.3a6-py2.7.egg/pyramid/config/__init__.py", line 61, in <module>
    from pyramid.config.assets import AssetsConfiguratorMixin
  File "/installation/dir/venv/local/lib/python2.7/site-packages/pyramid-1.3a6-py2.7.egg/pyramid/config/assets.py", line 83, in <module>
    @implementer(IPackageOverrides)
  File "/installation/dir/venv/local/lib/python2.7/site-packages/zope.interface-3.8.0-py2.7-linux-x86_64.egg/zope/interface/declarations.py", line 480, in __
    classImplements(ob, *self.interfaces)
  File "/installation/dir/venv/local/lib/python2.7/site-packages/zope.interface-3.8.0-py2.7-linux-x86_64.egg/zope/interface/declarations.py", line 445, in cl
    spec = implementedBy(cls)
  File "/installation/dir/venv/local/lib/python2.7/site-packages/zope.interface-3.8.0-py2.7-linux-x86_64.egg/zope/interface/declarations.py", line 285, in im
    spec = cls.__dict__.get('__implemented__')
RuntimeError: class.__dict__ not accessible in restricted mode

Ubuntu Precise, 64bit, with latest Apache, mod_wsgi, Python 2.7, using mpm_worker + mod_wsgi in daemon mode. This is the only program running on the server and there is only one wsgi interpreter in the config. Is this because of mpm_worker spawning new threads or what? More importantly – how do we fix it.

We have the following to subdivide requests to 4 daemon processes based on a cookie.

WSGIPythonOptimize 1

WSGIDaemonProcess sticky01 processes=1 threads=16 display-name=%{GROUP}
WSGIDaemonProcess sticky02 processes=1 threads=16 display-name=%{GROUP}
WSGIDaemonProcess sticky03 processes=1 threads=16 display-name=%{GROUP}
WSGIDaemonProcess sticky04 processes=1 threads=16 display-name=%{GROUP}

<VirtualHost *:81>
    ...
    WSGIRestrictProcess sticky01 sticky02 sticky03 sticky04
    WSGIProcessGroup %{ENV:PROCESS}
    ...

    WSGIScriptAlias / /installation/dir/our-program/prod-dispatch.wsgi        
</VirtualHost>

Answer 1

It has been known for ages that multiple subinterpreters don’t play well along C extensions. However, what I did not realize is that the default settings are very unfortunate. ModWSGI wiki clearly states that the default value for WSGIApplicationGroup directive is %{RESOURCE} the effect of which shall be that

The application group name will be set to the server hostname and port
as for the %{SERVER} variable, to which the value of WSGI environment
variable SCRIPT_NAME is appended separated by the file separator
character.

This means that for each Host: header ever encountered while accessing the server the mod_wsgi kindly spawns a new subinterpreter, for which the C extensions are then loaded.

I had unknowingly triggered the error by accessing localhost.invalid:81 with links browser on this local server causing 1 of our 4 WSGIDaemonProcesses to fail for all future incoming requests.

Summa summarum: always when using mod_wsgi with pyramid or any other framework that uses C extensions, make sure that WSGIApplicationGroup is always set to %{GLOBAL}. In other words, the result of using the default settings will cause you to shoot yourself in the foot, after which you might want to shoot yourself in the head too.