Home/ Questions/Q 223365
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T19:12:52+00:00

I read some articles on Web crawling and learnt the basics of crawling. According

  • 0

I read some articles on Web crawling and learnt the basics of crawling. According to them, the web crawlers just use the URLs retrieved by the other web pages and going through a tree (a mesh practically)

In this case how does a crawler ensures the maximum coverage. Obviously there may be a lot of sites that don’t have referral links from other pages/sites. Does the search engines follow any other mechanisms other than crawling and manual registration? (i.e getting information from domain registries)

If they are just based on crawling, How should we select a good set of “Root” sites to begin crawling? (We have no way to predict the results. If we select 100 sites with no referel links the engine will come up with just 100 sites + their inner pages)

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Obviously there may be a lot of sites
    that don’t have referral links from
    other pages/sites.

    I don’t think this really is as big a problem as you think.

    Does the search engines follow any
    other mechanisms other than crawling
    and manual registration? (i.e getting
    information from domain registries)

    None that I heard of.

    If they are just based on crawling,
    How should we select a good set of
    “Root” sites to begin crawling?

    Any kind of general-purpose web directory like the open directory project would be an ideal candidate, as would social bookmark sites like Digg or del.icio.us

    • 0