Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have build a website in Django. I need to use the web crawling features. So I installed Scrapy. Scrapy is working, as stated in their tutorial, by using
scrapy startproject dmoz
But that works on a diff folder with its own files.
Is it possible to use Scrapy functions in Django classes and put data in Django variables?
Crawling is a long, time consuming task. It doesn’t mix well with short HTTP requests. Therefore, you should use a user interface (presumably written in Django) to fire off Scrapy crawls, and a shared storage to monitor the state of the crawl.
But you wouldn’t want to crawl during the lifetime of a HTTP requests. That’s not how Scrapy is meant to be used.