Question

I have written a very basic crawler which scrapes product information from websites to be put into a database.

It all works well except that some sites seems to have a distinct URL for multiple parts of the page. For example, a product url might be:

http://www.example.com/product?id=52

then, it might have another URL for different parts such as comments etc:

http://www.example.com/product?id=52&revpage=1

My crawler is seeing this as a distinct URL. Ive found some sites where one product has hundreds of distinct URLs. Ive already added the logic to ignore anything after a hash in the url to avoid anchor’s, but I was wondering if anyone had any suggestions to avoid this problem? There could be a simple solution im not seeing.

At the moment its slowing down the crawl/scrape process where a site might have only 100 products its adding thousands of URLs.

I thought about ignoring the querystring, or even certain parts of the querystring but the product id is usually located in the query string so I couldn’t figure out a way, without writing an exception for each site’s URL structure

Answer 1

To elaborate on my comment…

You could include the following code

$producturl //is the url where you first found a product to scrape
$nexturl //is the next url you plan to crawl
if (strpos($nexturl, $producturl) === false) {
    crawl
}
loop back to the next url...

I am guessing you are crawling in sequence… meaning you find a page and crawl to all the links from that page… then you go back one level and repeat… If you are not crawling in sequence, you could store all the pages where you found a product and use that to check if the new page you plan to crawl starts with an url you have already crawled. If yes, you don’t crawl the new page.

I hope this helps. Good luck!