Question

Needed content is located on the same page with a static URL.

I created a spider that scrapes this page and stores the items in CSV. But it does so only once and then finish the crawling process. But I need repeat the operation continuously. How can I do this?

Scrapy 0.12

Python 2.5

Answer 1

Well giving you a specific example is kind of tough because I don’t know what spider you’re using and the internal workings of it, but something like this could work.

from scrapy.http import Request

class YourSpider(BaseSpider):
    # ...spider init details...
    def parse(self, response):
        # ...process item...
        yield item           
        yield Request(response.url, callback=self.parse)