Question

I have removed MacPorts from my Mac (10.7.5) and now Python seems to be broken. I am trying to run scrapy and I get the following error at the end:

ImportError: dlopen(/Library/Python/2.7/site-packages/lxml-2.3.4-py2.7-macosx-10.7-intel.egg/lxml/etree.so, 2): Symbol not found: _exsltDateXpathCtxtRegister
  Referenced from: /Library/Python/2.7/site-packages/lxml-2.3.4-py2.7-macosx-10.7-intel.egg/lxml/etree.so
  Expected in: /usr/lib/libexslt.0.dylib
 in /Library/Python/2.7/site-packages/lxml-2.3.4-py2.7-macosx-10.7-intel.egg/lxml/etree.so

When I run python and try to import that library, I get the same error as following:

$ python
Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 14:13:39) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Library/Python/2.7/site-packages/lxml-2.3.4-py2.7-macosx-10.7-intel.egg/lxml/etree.so, 2): Symbol not found: _xsltDocDefaultLoader
  Referenced from: /Library/Python/2.7/site-packages/lxml-2.3.4-py2.7-macosx-10.7-intel.egg/lxml/etree.so
  Expected in: flat namespace
 in /Library/Python/2.7/site-packages/lxml-2.3.4-py2.7-macosx-10.7-intel.egg/lxml/etree.so

How can I clean up this mess now?

Answer 1

Assuming this is the system Python, and you installed lxml into it via pip, and the version of libxml2 that comes with 10.7.5 is good enough for you (I think they started shipping somewhat recent versions in 10.7—as in 2.7.x—but I can’t remember for sure), it should be as simple as:

sudo pip uninstall lxml
sudo pip install lxml

The problem here is that lxml was built against MacPorts libxml2/libxslt2, instead of against your system versions. Theoretically it might be possible to just repoint the dependencies with install_name_tool, but that’s more likely to fail miserably—or, if you’re really unlucky, to seem to work but then crash when you really need it… Just re-building lxml should fix the problem. And if you used pip, that’s as easy as uninstall/reinstall.

If you didn’t use pip, re-doing the manual sudo python setup.py install from a clean copy of the tarball, or re-doing the sudo easy_install lxml, or doing a sudo pip install lxml over top of what you already have, will probably work, but that isn’t guaranteed.

If I’m wrong about Lion coming with decent libxml2, please say so in a comment, and I’ll give you the options for getting it (without reinstalling MacPorts).

If you want to be really sure you’ve cleaned everything, you can always restore your system site-packages to their default state just by doing this:

sudo rm -rf /Library/Python/2.7/site-packages/*
rm -rf ~/Library/Python/2.7/site-packages/*

Then, of course, you’ll have to reinstall anything you installed. (If you didn’t use pip last time, start with sudo easy_install pip and then use pip for everything else…) You can get a pretty good idea of exactly what you have installed just by doing an ls on those same directories, but that won’t necessarily tell you how you installed everything. (For example, I’ve got a couple packages I installed with pip install git+https://some.site/some/repo, or from a binary installer, or in one case I have absolutely no idea…) So, consider this the nuclear option if you’ve got no other choice—not a first resort, but better than the thermonuclear option of reinstalling OS X.