Question

How can I scrape a site using a User-Agent for Ipad?

I have this code below using curl in PHP which outputs the source but can’t find the tags still. On Ipad or Safari browser using an Ipad User-Agent, the tags displays when the site is loaded.

Thanks!

<?php
    $useragent= "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10')";

    $ch = curl_init ("http://www.cbsnews.com/video/watch/?id=7370279n&tag=mg;mostpopvideo");

    curl_setopt ($ch, CURLOPT_USERAGENT, $useragent); // set user agent
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
    // curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
    echo $output = curl_exec ($ch);

    curl_close($ch);
?>

Answer 1

Try using curl from the command line, with a perl script such as this:

my $ua = "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10";
my $curl = "curl -A '$ua'";
my $server = "http://www.cbsnews.com";
my $startpage = "$server/video/watch/?id=7370279n&tag=mg;mostpopvideo";
my $path = "/path/to/download/to";
open(f, "$curl -L $startpage |") or die "Cannot open website: $!";
while (<f>)
{
    if (/<a\s+[^>]*href=\"$server\/([^\"\/])*\"/)
    {
        my $file = $2;
        system("$curl -e $startpage $server/$file > $path/$file");
        next;
    }

    if (/<a\s+[^>]*href=\"$server\/([^\"]+)\/([^\"\/])*\"/)
    {
        my $folder = $1;
        my $file = "$folder/$2";
        system("mkdir -p $path/$folder");
        system("$curl -e $startpage $server/$file > $path/$file");
        next;
    }
}
close(f);