Home/ Questions/Q 6901003
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T07:37:07+00:00

I wrote a basic CrawlSpider in scrapy but I want to understand what is

  • 0

I wrote a basic CrawlSpider in scrapy but I want to understand what is the order in which the urls are crawled – FIFO / LIFO?

I want that the crawler should crawl all the links in the start url page and then move on to other URLs which does not seem to be the order.

How can I do this?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    http://readthedocs.org/docs/scrapy/en/0.14/faq.html#does-scrapy-crawl-in-breath-first-or-depth-first-order

    By default, Scrapy uses a LIFO queue for storing pending requests,
    which basically means that it crawls in DFO order. This order is more
    convenient in most cases. If you do want to crawl in true BFO order,
    you can do it by setting the following settings:

     DEPTH_PRIORITY = 1
 SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
 SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
    • 0