Question

I’m trying to run Hadoopy, but am getting a compiling error on OS X:

ImportError: Building module failed: ["CompileError: command 'llvm-gcc-4.2' failed with exit status 1\n"

I have /Developer/usr/bin in my $PATH, and am running latest version of XCode on OS X Lion 10.7. Cython was installed via easy_install.

Full output:

>>> import pyximport; pyximport.install()
>>> import hadoopy
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:236:22: error: getdelim.h: No such file or directory
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:236:22: error: getdelim.h: No such file or directory
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c: In function ‘__pyx_f_7hadoopy_5_main_11HadoopyTask_read_offset_value_text’:
/Users/dolan/.pyxbld/temp.macosx-10.7-intel-2.7/pyrex/hadoopy/_main.c:4399: warning: implicit conversion shortens 64-bit value into a 32-bit value
lipo: can't open input file: /var/folders/8b/n0j5pn_13qn_x8p2v4f848zh0000gn/T//ccC8x2Ex.out (No such file or directory)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/hadoopy/__init__.py", line 22, in <module>
    from _main import run, print_doc_quit
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 335, in load_module
    self.pyxbuild_dir)
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 183, in load_module
    so_path = build_module(name, pyxfilename, pyxbuild_dir)
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyximport.py", line 167, in build_module
    reload_support=pyxargs.reload_support)
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/pyximport/pyxbuild.py", line 85, in pyx_to_dll
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/Cython/Distutils/build_ext.py", line 135, in run
    _build_ext.build_ext.run(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/Library/Python/2.7/site-packages/Cython-0.15.1-py2.7-macosx-10.7-intel.egg/Cython/Distutils/build_ext.py", line 143, in build_extensions
    self.build_extension(ext)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
    depends=ext.depends)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 624, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/unixccompiler.py", line 180, in _compile
    raise CompileError, msg
ImportError: Building module failed: ["CompileError: command 'llvm-gcc-4.2' failed with exit status 1\n"

Answer 1

Instead of using pyximport, build the extension modules in place with python setup.py build_ext --inplace (or install for in-place development with python setup.py develop, or just a regular install via python setup.py install). For packages you almost always want to run the setup, which will properly configure the compilation environment, build, and installation process.

---

pyximport is good for your personal scripts if you’re using Cython to speed up your code (e.g. for scientific computing). Even then you might need to bring in other libraries, and build from multiple sources. In that case, you can use a pyxbld file to set up the sources and include_dirs. For example, say you have foo.pyx, then you can place build instructions in a foo.pyxbld in the same directory. For example:

#foo.pyxbld
def make_ext(modname, pyxfilename):
    from distutils.extension import Extension
    return Extension(name = modname,
                     sources=[pyxfilename, 'bar.c'],
                     include_dirs=['/myinclude'] )
def make_setup_args():
    return dict(script_args=["--compiler=mingw32"])

Now using foo is as simple as the following:

import pyximport
pyximport.install()
import foo

Presuming there’s no compilation errors, it’s virtually transparent but for the delay the first time you import the module. Subsequent imports will look for the built extension in the .pyxbld subdirectory of your home/profile directory.