Question

tl,dr: I have buildouts that work fine with “bin/buildout” from the shell, but fail when I run subprocess.call([“bin/buildout”]) or similar from Python. Why? Any workaround?

I have added the ‘python’ tag instead of just ‘buildout’ because this may be a subtle difference in calling a python script from the shell vs from python using subprocess.call() or os.system(). I don’t know why they would be different. It’s probably a Buildout thing though because Buildout rewrites itself then restarts itself.

To construct the example I want to show, I start with a new Ubuntu 12.04 LTS virtual box. Then I install git on it (sudo apt-get install git) and clone one of our repositories that has hardly anything in it:

git clone git://github.com/lizardsystem/lizard-datasourceviewer.git

Then I cd into it and run bootstrap.py:

cd lizard-datasourceviewer
python bootstrap.py

So far, so good. Now it is possible to run “bin/buildout”, and it will run without trouble (well, at some point it errors out because the system doesn’t have matplotlib — that’s the expected result). But don’t do that, because once you do that, the error below won’t occur. If you did, delete the directory and clone it again.

If INSTEAD of that I run it from Python, like:

$ python
>>> import subprocess
>>> subprocess.call(["bin/buildout"])

Then this fails much earlier (see below for the error). Which is a problem because we want to call buildout from a script. Variations of the subprocess call, like os.system(“bin/buildout”) or subprocess.call([“/bin/sh”, “-c”, “bin/buildout”]) don’t help.
Once bin/buildout has run from the command line once, the problem is gone, even if bootstrap.py is called again.

I know when it goes wrong. Initially, bin/buildout looks like this:

#!/usr/bin/python

import sys
sys.path[0:0] = [
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/usr/lib/python2.7/dist-packages',
  '/home/vagrant/lizard-datasourceviewer/eggs/zc.buildout-1.4.4-py2.7.egg',
  ]

import zc.buildout.buildout

if __name__ == '__main__':
    zc.buildout.buildout.main()

And after running bin/buildout from the command line, it gets “distribute”, rewrites bin/buildout, and restarts itself. As a result bin/buildout looks like this:

#!/usr/bin/python

import sys
sys.path[0:0] = [
  '/home/vagrant/lizard-datasourceviewer/eggs/zc.buildout-1.4.4-py2.7.egg',
  '/home/vagrant/lizard-datasourceviewer/eggs/distribute-0.6.27-py2.7.egg',
  ]

import zc.buildout.buildout

if __name__ == '__main__':
    zc.buildout.buildout.main()

It is this rewrite and restart step that seems to fail when bin/buildout is run from Python instead of from the shell. The error message is:

vagrant@precise64:~/lizard-datasourceviewer$ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.call(["bin/buildout"])
Getting distribution for 'mr.developer==1.21'.
Got mr.developer 1.21.
Getting distribution for 'buildout-versions==1.5'.
Got buildout-versions 1.5.
mr.developer: Creating missing sources dir /home/vagrant/lizard-datasourceviewer/src.
Getting distribution for 'distribute==0.6.27'.
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /usr/lib/python2.7/dist-packages
Non-egg installation
Removing elements out of the way...
Already patched.
/usr/lib/python2.7/dist-packages/setuptools.egg-info already patched.
After install bootstrap.
Don't have permissions to write /usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info, skipping
Creating /usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info
**error: /usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info: Permission denied**
An error occured when trying to install distribute 0.6.27. Look above this message for any errors that were output by easy_install.
While:
  Installing.
  Checking for upgrades.
  Getting distribution for 'distribute==0.6.27'.
Error: Couldn't install: distribute 0.6.27
1

As you can see, it’s trying to install into the system dist-packages where this user doesn’t have access. But WHY? What’s the difference with running the same script from the shell?

Answer 1

Not sure why, but using subprocess.call(['/bin/bash', '-c', 'bin/buildout']) seemed to fix it for me, where subprocess.call(['/bin/sh', '-c', 'bin/buildout']) failed. Ubuntu 12.04.

Hope it helps. 🙂