Question

I am trying to program a web-crawler, but now I am wondering: What’s the best method to store all the urls, so that crawlers can work together but do not interfere?

Example:

• Crawler 1 finds a page with 100 URLs
• Crawler 2 finds a page without any URLs
• Crawler 1 and 2 shall share the 100 URLs Crawler 1 has found

My ideas (two different approaches)

1. Let the Crawler scan the page for new URLs
   • save all found URLs in a Queue (PriorityQueue) shared by all instances of Crawler
   • outsource the URLs to a database as soon as the Queue becomes too large (e.g. 80% of the maximum size)
2. Let the Crawler save HTML and an Analyzer search for URLs later
   • save new URLs to a horizontally scaled database
   • always ask this database for the next URL

Problems

1. Is the shared Queue a bottle neck?
2. How can I prevent multiple crawlers from getting the same URL twice if database is only eventually consistent?

Other solutions?

Are there any other solution? What is the standard solution for web crawlers?

Answer 1

You have more than one question there, but how about this for one general approach.

See this SO discussion regarding determination if two URLs are equivalent.

Then, you really have a lot of options. My personal favorite would be to write 3 things.

1) A database application that only handles the list of URL’s waiting to be visited.
2) A database application that stores important results from visited sites. Whatever you want to save.
3) An application that can query and post to (1) and then post to (2)

Why this way?
You can place (1) and (2) on a single machine. (3) can run from anywhere, allowing you to run multiple instances from multiple IP addresses. This will help you navigate service providers and other Network folks, who may be perturbed by your repeated and frequent http requests.