Question

I wrote a program in PHP to find and print all links present on a web page. It also goes inside any links it found and does the same. My problem is that in some sites (like Youtube) it won’t print the links, or follow inside them.

Here is my main code:

function echo_urls($site_address){

    if(check_valid_url($site_address)){
        $site = new site();
        $site->address = $site_address;
        $site->full_address = "<a href=\"$site_address\">$site_address</a>";
        $site->depth = 0;

        $queue = new queue();

        $queue->push($site);
        array_push($queue->seen,$site->address);

        $depth = 0;

        while(($site = $queue->get_first())){
            $depth++;
            echo $site->depth." : ".$site->full_address."<br>";
            $queue = push_links($site->address,$queue,$depth);
        }
    }
    else;
}
function push_links($site_address,$queue,$depth){
if($depth<4){
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL,$site_address);
        curl_setopt($ch, CURLOPT_TIMEOUT, 30); //timeout after 30 seconds
        curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
        $result=curl_exec ($ch);
        curl_close ($ch);

        if( $result ){
            preg_match_all( '/<a\s[^>]*href=([\"\']??)([^\" >]*?)\\1[^>]*>(.*)<\/a>/siU', $result, $list);

            $list = $list[0];

            foreach( $list as $item ) {
                if(!(empty($item)))
                if($result = get_all_string_between($item,"href=\"","\"")){
                    if((array_search($result[0],$queue->seen))==false){
                        $site = new site();
                        $site->address = $result[0];
                        $site->full_address = $item;
                        $site->depth = $depth;
                        $queue->push($site);
                        array_push($queue->seen,$site->address);
                    }
                }
            }
        }
    }

    return $queue;
}

Answer 1

It’s hard to tell by looking at a couple of functions, but my guess is:

1. YouTube is blocking you
2. This part if($depth<4){ is stopping push_links from executing because it might be returning FALSE

Also, don’t use RegEx for this. Use something like The DOMDocument class