Question

I’m considering developing a site where the server will crawl another site periodically, in order to gather content for certain entries in my database. My quesitons are as follows…

1. How do you get the server to execute a crawl timely?
2. Can you get it to execute php or what language do you use to perform the crawl?
3. Are there any good APIs to do this?
4. Should I consider building my own? If so, some advice on how to get started would be great

Basically, the kind of thing I want to do, is for the server to execute a script (say every hour), which finds all entries in the database which haven’t yet been crawled on another site. It will take a certain value from those entries, and will use them to crawl another site… it might request a url like this: www.anothersite.com/images?q=entryindb.

What I want it to do is then crawl the HTML, return an array, and log the values in the database. This is what I want the crawler to look for

Find all instances of 
<img> inside <a> inside <td> inside <tr> inside <tbody> inside <table> inside <div id='content'>
Return array of the img.src from all instances.

Is something like that possible? – If so, how would I go about doing it? – Please bear in mind that web dev wise, the only experience I have so far (server-side) is with PHP.

UPDATE: I will be using a linux-based server, so I guess chron-scripting is how I should do it?

Answer 1

1. You can use cron
2. Yes, you can run a PHP script
3. Nothing like a complete crawling API (AFAIK), but there are classes which will help you parse and traverse DOM documents.
4. You can set something up in minutes if you follow the following steps

1. You need phpQuery to make your life easier with this

Download phpQuery-0.9.5.386-onefile.zip from here.

2. Your PHP file would be something like this

require_once 'phpQuery-onefile.php';
$html = file_get_contents('http://www.othersite.com');
phpQuery::newDocumentXHTML($html);

$elements = pq('#content table tbody tr td a img');
$images = array();
foreach($elements as $img){
    $images[] = pq($img)->attr('src');
}

The $images array will have a list of all the image sources.

3. Save the above code in a file, say crawler.php

Then in the cron tab, if you want the crawler to run every hour, you would do:

0 * * * * php /path/to/your/crawler.php