Home/ Questions/Q 8247345
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-07T22:49:06+00:00

My problem is to crawl every page and every document starting from a certain

  • 0

My problem is to crawl every page and every document starting from a certain list of seed.

I have installed nutch and make it run with the following commmand:

bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5

I expected the nutch process to crawl something like 100 url, but it says it found just 11 document. So i tried to run nutch with this command:

bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 4

and it found 23 document.

I’m running the process starting form the test seed http://nutch.apache.org

Why nutch has this behavior? How can I set nutch to crawl every url starting from my seeds in a certain depth?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    topN sets the number of url to be fetched in every depth. In your first example depth is 3. Depth1 is seed url. And in depth2 and depth3, 5(topN value) urls will be fetched. 5*2 (depth2 and depth3) + 1 (seed url i.e depth1) = 11. TO fetch more urls you can increase topN. If you do not want to restrict then you can skip topN argument.

    • 0