Question

I have a little tool written in JavaScript and PHP that takes a list of URLs and checks HTTP status codes for all of them.
I use curl to check the actual status.
It works great as long as I have nice URLs. I’m having a problem with a URL that has ® in it. My tool returns 404 when I know it should return 301.

My guess is that this ‘®’ is being converted to something like %C2 and causes a problem.

I know it can be done because pasting this same URL here returns 301 as it should.

My PHP curl looks like this:

        ...
        if (($curl = curl_init()) == false) {
            throw new Exception('curl_init error for url '.$_POST['url'].'.');
        }
        $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
        $header[] = "Cache-Control: max-age=0";
        $header[] = "Connection: keep-alive";
        $header[] = "Keep-Alive: 300";
        $header[] = "Accept-Charset: iso-8859-1,utf-8;q=0.7,*;q=0.7";
        $header[] = "Accept-Language: en-US;q=0.5";
        $header[] = "Pragma: ";
        
        curl_setopt($curl, CURLOPT_URL, $_POST['url']);
        curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
        curl_setopt($curl, CURLOPT_NOBODY, true);
        curl_setopt($curl, CURLOPT_AUTOREFERER, true);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl, CURLOPT_TIMEOUT, 50);
        $Cresponse = curl_exec($curl); // execute the curl command
        $response['callback']['data'] = $http_status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
        curl_close($curl);
        ...

I tried to use urldecode() but this encodes the whole URL along with http:// to http%3A%2F%2F.

Any idea why this ® is causing problems?

Answer 1

Use parse_url() and urlencode() just the path, query and fragment.

Then reassemble the encoded URL and make the request.

$url = parse_url ($_POST['url']);
if ($url === FALSE) {
    /* error handling */
}
$encoded_url = $url['scheme'] . "://" .
               $url['host']   .
               urlencode ($url['path])     . "?"   .
               urlencode ($url['query])    . "#"   .
               urlencode ($url['fragment])