Home/ Questions/Q 3317002
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-17T22:32:10+00:00

I used the example on installing nutch from their wiki. I was able to

  • 0

I used the example on installing nutch from their wiki. I was able to crawl multiple pages pulled from dmoz easily. But is there a configuration that can be done to crawl external links it finds on a page, or write those external links to a file to be crawled next?

What is the best way to follow links on a page to index that page as well with nutch? If I were executing the bin/nutch via python, could I get back all the external links it found, and create a new crawl list to run again? What would you do?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    First, make sure that the parameter ‘db.ignore.external.links’ is set to false. Also, in the file ‘regex-urlfilter.txt’, add rules for the external links you wish to be crawled OR add +. as the last rule. The +. rule will make the crawler follow ALL links. If you use that last option, beware that you risk crawling all the Web!

    • 0