Question

I am trying to obtain data from a Web and show it to the user using cURL and Simple HTML Dom PHP class.

Some pages have a redirection depending on the client’s language, I am using a function to determine the final page that is to be scraped.

In order to show it as the user would see it, I am using this:

$useragent = $_SERVER['HTTP_USER_AGENT'];
curl_setopt($ch, CURLOPT_USERAGENT,  $useragent);

At the moment most of my current users are Spanish speakers, therefore I am temporarily limiting accepted languages so if there is a language redirect on the target page, it will show Spanish or English first.

$header[] = "Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3";

However, since my server is located in the Netherlands and some pages have an IP-based redirector, sometimes the pages redirect to the /nl/ directory, ignoring the language parameters.

This happens, for example, with the http://www.econsultancy.com Website.

Is it possible to avoid this kind of redirect, maybe using the client’s IP address in the cURL request?

Also, is it possible to use the client’s browser language settings to make the Accept-Language parameter dynamic?

Here’s the entire function script:

<?
function redirector($originalurl) {

$ch = curl_init();

$useragent = $_SERVER['HTTP_USER_AGENT'];

    $header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
    $header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3";
    $header[] = "Pragma: ";

curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_USERAGENT,  $useragent); 
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt($ch, CURLOPT_URL, $originalurl);
    $out = curl_exec($ch);

    $out = str_replace("\r", "", $out);

    $headers_end = strpos($out, "\n\n");
    if( $headers_end !== false ) { 
        $out = substr($out, 0, $headers_end);
    }   

    $headers = explode("\n", $out);
    foreach($headers as $header) {
        if( substr($header, 0, 10) == "Location: " ) { 
            $target = substr($header, 10);

            $targeturl = $target;
        }   
    }

return $targeturl;
}
?>

Thanks in advance!

Answer 1

Some of IP based redirections are really stubborn (and it’s almost impossible to switch certain pages to English from <whatever page thinks your language is>), but you may try to intercept any redirection by using CURLOPT_FOLLOWLOCATION set to False and parsing Location header (this solution requires you to guess URL correctly):

$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if( $status_code >= 300 && $status_code < 400){
    // Was redirect, parse header
    if( substr( $location, '/nl/') !== false){
         // Try to load page $url = str_replace( $location, '/nl/', '/en/');
         // Remember to check whether page exists
    } else {
         // Continue request
    }
}

Edit – per site

If you can afford to do this on “per site” basis (to create function for each site to switch language) you may trace what’s happening when you are switching languages (for example Firefox has perfect plugin for this) and most of the time you’ll end up using:

• hardcored (user has no control what so ever about language) – you’re screwed
• different URLs (we’ve already covered that) – you’ll see /nl/, lang=nl, l=nl, … in the URL
• language is stored inside the cookies – you just need to set the cookie once [tutorial] and then it’ll just be correct
• language change is available for registered users – you need to create user on each site, switch his (its) language to <whatever you want> and then (when opening page) you first simulate logging in on the page (using cookies open login form and POST username and password)

With a little luck you’ll be good with what you’re having already in combination of “large array” of cookie values pair like this:

array (
    'lang' => 'eng',
    'l' => 'en',
    'Language' => 'Engligh',
    ...
)

But once you encounter two pages which uses the same “cookie variable name” and different values:

lang=eng
lang=en
lang=7

You’re screwed and you’ll have to use some sort of switch($domain) again.