Question

I am having problems reading some urls. There is nothing wrong with the urls, as I can view them in my browser (an example of one such URL is given below):

http://www.bloomberg.com/apps/news?pid=20601087&sid=a2BhXFMpbb5M

I am using fopen like this in my code:

public static function grokPage($path)
{
    $data = '';
    $file = fopen($path, "r");

    if ($file)
    {
        while (!feof($file))
            $data .= fgets($file, 1024);
    }
    return $data;
}

the error I get is:

Warning: fopen(http://www.bloomberg.com/apps/news?pid=20601087&sid=a2BhXFMpbb5M) [0function.fopen0]: failed to open stream: Redirection limit reached, aborting in xxx_filename.php

From the PHP fopen doc, it seems I am using the function correctly. Does anyone understand the recursion warning and how to fix it?

Answer 1

This means that your target page is returning more redirects to different addresses (probably using a Location: header) than your max_redirects setting has specified.

This looks like a very good article on how to fetch web pages using the fopen wrappers. It contains an example on how to change the max_redirects setting.

It could well be, though, that Bloomberg are shutting you out intentionally because it detects automated data scraping, which may be a violation of their terms and conditions.