Home/ Questions/Q 7737499
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-01T08:00:55+00:00

I want to build a dataset consisting about 2000-3000 web pages, starting with several

  • 0

I want to build a dataset consisting about 2000-3000 web pages, starting with several seed URLs. I tried it using the Nutch crawler but I was unable to get it done (unable to convert the ‘segments’ data fetched into html pages) .

Any suggestions of a different crawler that you have used or any other tool? What if web pages contain absolute URLs which will make offline use of the dataset impossible?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You can NOT directly convert the nutch crawled segments to html files directly.

    I suggest you these options:

    1. You can try modifying the source code to do that. (study the org.apache.nutch.segment.SegmentReader class. You can then dig into it to modify the working as per your use case).
    2. EASY SOLUTION if you dont want to invest time to study code: Use nutch to crawl all required pages. Then get the actual urls crawled by using the “bin/nutch readdb” command (use dump option). Then write a script to wget the urls and save it in html form. Done !!
    • 0