Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I have a few desktop machines at different geographical locations. I need to create a crawler with clients at each desktop machine and a central server where data is indexed. Is it possible to create such a crawler in Nutch? Are there any alternatives. Python based crawlers would be preferable.
If you use Nutch like buffer suggested, there is a script on the Nutch Wiki that may be able to help you. You would just need to get the linkdb, crawldb, and segments from each system to the central server before doing this – I think attempting to access those resources remotely would take a long time during the indexing process.