Question

I have been running a script on my ErrorDocument pages that logs and emails me when the referrer is my site. Recently I have thought I could use the same script to log broken external links also. I’m attempting to retrieve the external pages headers, if they show a 404 stay on my site with an error message, and if not then do the redirect. This is working but it is very slow even when the site is available. Much slower than accessing the external page directly.

I think the problem must lie in how I am retrieving the headers (see function is404) rather than in the error reporting or all the preg_replace‘s, as it works quite quickly when I use it in my ErrorDocument‘s, but I will post all the code below anyway in case there are issues elsewhere.

Aside:
I haven’t decided whether or not to deploy this yet, obviously it’s dependent on getting the speed issue fixed, but I wonder what your opinions are on this approach assuming it works up to speed. The benefit for my site are clear, it keeps users on the site rather than dumping to a possibly indecipherable 404 and it keeps me informed of broken links. If the external page has a custom 404 however, perhaps the user would be better off following the broken link. I’ve tried to offset this, by including a link to the 404 in the error message and also checking the domain for availability and providing a link to that too if it’s available. Other things are feasible too, links to google results for example. So, if you came across this would you be pleased, ambivalent, mildly annoyed, or disgusted?

Update:
I have decided in the end to go for a framed solution, as suggested by Mel, whereby I load the external page in an iframe and have a small bar across the top that allows users to report broken links should they exist. The problem that lied in this solution was that if I didn’t actually redirect the page, if the user wished to bookmark it they’d end up with a link to my redirection script (which is blocked unless the referer is my site; so they’d have a link to a forbidden page). This is fixed by a simple window.location=http://external.site, but then the issue is when to call it. The iframe onload event fires whether the page is a 404 or not so that was no good and using a simple timeout made pages that had clearly already loaded just look ugly with my “has this page error’d” message across the top. I have gone for a combination. When onload fires I am making an Ajax call to a script that checks the status of the external site from my server. (I’m using the same is404 method that I had below but it seems to run perfectly fine this time around. I guess the speed issue was elsewhere, although I’m not sure where.) I think using an Ajax call (with a timeout) will prevent the site blocking problem pointed out by Adrian. If my server thinks the page is not a 404, I remove the banner immediately. If it thinks it is I set a timeout and leave the banner up (hopefully) long enough for users who want to to click on it. There will be cases, as pointed out in the answers, where my 404 code doesn’t match with what the user sees but the worst cases here are a missed opportunity to report the error or a banner that stays up for a while on a fine page. Hopefully the majority of the time it will work out. For users that don’t have javascript enabled I just use a meta refresh and immediately forward to the link. If I’ve got something wrong here or missed something else important I’d appreciate your feedback. Thanks.

---

Here’s the code:

  //  /../php/urlfns.php

  1 <?php
  2 function hasProtocol($uri) {
  3     return preg_match('#^.+://#', $uri);
  4 }
  5 
  6 function getDomain($uri, $keepproto=false) {
  7     return preg_replace('#^((.+://)?(.+?))\/.*#', $keepproto?'${1}':'${3}', $uri.'/');
  8 }
  9 
 10 function getBaseDomain($uri, $keepproto=false) {
 11     $dom = getDomain($uri, $keepproto);
 12     if(!$keepproto) {
 13         return preg_replace('#.*\.(.+\..+)$#', '${1}', '.'.$dom);
 14     }
 15     else
 16     if(hasProtocol($dom)) {
 17         return preg_replace('#^(.+://).*\.(.+\..+)$#', '${1}${2}', $dom);
 18     }
 19     else {
 20         return preg_replace('#^.*\.(.+\..+)$#', '${1}', '.'.$dom);
 21     }
 22 }
 23 
 24 function isOnDomain($uri, $domain, $allowsubs=false) {
 25     $uridom = getDomain($uri);
 26 
 27     if(!$allowsubs) {
 28         return ($uridom===$domain);
 29     }
 30     else {
 31         $basedom = getBaseDomain($domain);
 32         return (strlen($uridom) - strlen($basedom) === strrpos($uridom, $basedom));
 33     }
 34 }
 35 
 36 function stripDomain($uri) {
 37     return preg_replace('#^(.+://)?.*?(/.*)#', '${2}', $uri);
 38 }
 39 
 40 function is404($uri) {
 41     stream_context_get_default(array(
 42         'http'=>array(
 43             'method' => 'HEAD'
 44         )
 45     ));
 46     $hds = @get_headers($uri);
 47     return (!$hds || strpos($hds[0], ' 404 ') !== false);
 48 }
 49 ?>

**

  //  /../php/badlink.php

  1 <?php
  2 require_once('urlfns.php');
  3 
  4 function reportbadlink($lntype='generic', $subdomains=false, $blcache='../badlinks/') {
  5 
  6     if(isset($_SERVER['HTTP_REFERER']) &&
  7         isOnDomain($_SERVER['HTTP_REFERER'], $_SERVER['SERVER_NAME'], $subdomains)) {
  8 
  9         $reffile = $_SERVER['DOCUMENT_ROOT'].'/'.stripDomain($_SERVER['HTTP_REFERER']);
 10 
 11         $blid  = md5($lntype.$_SERVER['REQUEST_URI'].$reffile.@filemtime($reffile));
 12         $blfil = dirname(__FILE__).'/'.$blcache.'/'.$blid;
 13 
 14         if(!file_exists($blfil)) {
 15             $report = '  On: '.$_SERVER['HTTP_REFERER']."\n".
 16                       '  To: '.$_SERVER['REQUEST_URI']."\n".
 17                       'Type: '.$lntype."\n".
 18                       '   #: '.$blid."\n\n";
 19 
 20             file_put_contents($blfil, $report);
 21             mail('webmaster@localhost', 'A broken link has been found.', $report);
 22         }
 23     }
 24 }
 25 ?>

**

  //  /ssi/extlink.php
  1 <?
  2     require_once('../../php/urlfns.php');
  3     $uri = $_SERVER['QUERY_STRING'];
  4 
  5     if(!hasProtocol($uri)) {
  6         $uri = 'http://'.$uri;
  7     }
  8     
  9     if(!is404($uri)) {
 10         header('Location: '.$uri);
 11         exit;
 12     }
 13         
 14     header("Cache-Control: no-cache, must-revalidate");
 15     header("Expires: Sat, 26 Jul 1997 05:00:00 GMT");
 16 
 17     require_once('../../php/badlink.php');
 18     reportbadlink('external');
 19         
 20     $baseuri = getDomain($uri,true);
 21     if(preg_match('#^'.$baseuri.'/?$#', $uri) || is404($baseuri)) {
 22         $baseuri = null;
 23     }
 24                
 25     $errtype = 'External page not found';
 26     $errmesg = '<p><em><a href="'.$uri.'">'.$uri.'</a> was not found.</em></p>'.
 27                '<p>This may be because the site is temporarily unavailable, or it may be a broken link.</p>'.
 28                ($baseuri ? '<p><a href="'.$baseuri.'">'.$baseuri.'</a> appears to be available however and you may find '.
 29                ' what you are looking for there.</p>' : '');
 30 ?>
 31 <!DOCTYPE html>
 32 <html lang="en" dir="ltr">
    ...

Answer 1

I definitely would not deploy this, simply because your assumption is that if you can reach the site then so can your user. However, if you can’t but the user can because of localized routing issues you have sent them a red herring.

I would rather deploy a framed solution, where in the small top frame you have a button to report the link as broken. Similar to the right pane of Google images. You can then keep a count in your db of how many times the link is reported broken in the last x days and display this next to a link.

For me the end user this would be much better, then somebody asserting that the site don’t work for me. “I don’t believe you, what shady things are you doing there, I’ll just copy the link, paste it in my address bar and go there myself!”. Then again, I’m a cynic.