Question

Trying to run scrapy crawl command using crontab under virtual environment, and getting below error while running scrapy command from cron job –

     UserWarning: Cannot import scrapy settings module myspider.settings warnings.warn("Cannot import scrapy settings module
 %s" % scrapy_module) 
     ..... 
     raise KeyError("Spider not found: %s" % spider_name) KeyError: 'Spider not found: myspider'

Any help or suggestion?

Answer 1

It was missing PYTHONPATH while running under crontab.

I have added it before my following cron job:

*/40 * * * * source /home/water/.virtualenvs/water/bin/activate && cd
$HOME/water2012/ && scrapy crawl water2012 >>
$HOME/water2012/log/log_$(date +\%Y\%m\%d).log 2>&1