Question

Could someone help us with a regular expression in order to detect repeated patterns inside a URL string? The goal is obviously to detect malformed weird URLs.

For example, the following URLs are alright:

http://www.somewhere.com/help/content/21/23/en/
http://www.somewhere.com/help/content/21/24/en/
http://www.somewhere.com/help/content/21/64/en/
http://www.somewhere.com/help/content/21/65/en/
http://www.somewhere.com/help/content/21/67/en/

While this this ones, are incorrect, and should be tagged:

http://www.somewhere.com/help/content/21/content/1/54/en/
http://www.somewhere.com/help/content/21/content/1/62/en/
http://www.somewhere.com/help/content/21/content/8/52/en/

Since content is repeated twice. So far we have been solving this using parse_url and explode, but it looks quite inefficient!

As well, I’m aware that there might be many URLs that repeat a number in the path, or some other value, so any suggestions to solve this issue would be more than welcome.

Thanks a lot!

For a better comprehension of the issue, you can visit the following link and click on “Administrador MySQL”:

http://www.elserver.com/ayuda/content/21/65/es/

Answer 1

Assuming you have a file (testdata.txt) which contains a list of URLs, one per line, the following tested script will extract those URLs having (at least) one repeated path segment:

<?php // test.php Rev:20120924_0800
$re = '%
    ^                  # Anchor to start of line.
    (?:[^:/?#\s]+:)?   # URI scheme (optional).
    (?://[^/?#\s]*)?   # URI Authority (optional).
    (?:/[^/?#\s]*)*?   # URI path segments (before repeats).
    /([^/?#\s]+)       # $1: Repeated URI path segment.
    (?:/[^/?#\s]*)*?   # URI path segments (between repeats)
    /\1                # $1: Repeated URI path segment.
    (?:/[^/?#\s]*)*    # URI path segments (after repeats).
    (?:\?[^#\s]*)?     # URI query (optional).
    (?:\#\S*)?         # URI fragment (optional).
    $                  # Anchor to end of line.
    %mx';
$text = file_get_contents('testdata.txt');
if (preg_match_all($re, $text, $matches)) print_r($matches[0]);
else echo("no matches!");
?>