Question

I’ve deployed a scrapy project, linked with a class which use MySQLdb (for persistence and get some data before run the spider)
On Python 2.7 (local machine), i haven’t problems. However, on Python 2.6 (in production) i get this error at each call of the crawler :

/usr/lib/python2.6/dist-packages/zope/__init__.py:3: UserWarning: Module _mysql was already imported from /usr/lib/pymodules/python2.6/_mysql.so, but /usr/lib/pymodules/python2.6 is being added to sys.path
import pkg_resources

I’ve tried to ignore warnings by warnings.filterwarnings('ignore'), but it has no effect. After few researchs, I believe the cause of this warning is beacause i have two same versions of MySQLdb in two directories of the pythonpath.
But, how to fix it ?

For your information, i have installed theses packages (Python 2.6), with aptitude :

python-mysql
python-scrapy0.14
python-lxml

There’s no way to update to Python 2.7 in production, sadly.

Thanks!

Answer 1

Problem solved, by reinstalling scrapy and MySQL-python with easy_install. What I have done :

apt-get remove --purge scrapy-0.14 python-mysqldb
apt-get autoremove
apt-get clean
easy_install -U scrapy
easy_install -U MySQL-python

Make sure that you have python-setuptools, python-dev and libmysqlclient-dev installed (via apt)