Question

I want to create custom search engine for multiple domain.
How can I use solr with nutch to create a custom search for 500+ domains, while searching each domain should be able to show its own data.

e.g.
example.com exapmle2.com example3.com and so on, When ever user searches on example.com he should get data which belongs to example.com same for example2.com and so on

these website may be blog post, e-commerce site, classified site or hotel reservation site.

any suggestion would be appreciated.

Answer 1

This should be possible right out of the box. When you index to solr using nutch schema it has a field called site that stores the domain. On the search interface(that you will build) when you select a domain (aka site) you just have to pass a filter query like “site:domain” so that the results are restricted to the domain searched.

NOTE: If you want to restrict crawls to the injected domains only make sure you set the external links property in nutch to false.

Hope that answers your question.