Question

How can I get the URLs containing supplied keywords from the website?

For example: I want to capture all of the anchor href’s on this page http://www.catererglobal.com/rzwritingajobad.html
containing any of the keywords (promote, job)

Expected results include:

http://www.catererglobal.com/recruiters/rz-promote-your-brand
http://www.catererglobal.com/recruiters/rz-job-advertising

Answer 1

This is how I do it in php =)

<?php
$oldSetting = libxml_use_internal_errors( true );
libxml_clear_errors();

$html = new DOMDocument();
$html->loadHtmlFile( 'http://www.catererglobal.com/rzwritingajobad.html' );
$xpath = new DOMXPath( $html );
$links = $xpath->query( '//a' );

foreach ( $links as $link ) {
    $cur = $link->getAttribute( 'href' );
    if (preg_match('/(promote|job)/', $cur)) { echo "$cur\n"; }
}

libxml_clear_errors();
libxml_use_internal_errors( $oldSetting );
?>

The output is :

http://www.catererglobal.com/recruiters/rz-job-advertising/10298792/post-a-job/
/recruiters/rz-job-advertising
/recruiters/rz-promote-your-brand
/moreterms/job-location
http://www.madgex.com/job-boards/

Xpath is our best friend 😉