Question

Function

function curlPost($url, $postArray = NULL, $timeout=2, $errorReport=FALSE) {
    # PREPARE THE POST STRING
    if ($postArray != NULL) {
        $postString = '';
        foreach ($postArray as $key => $val) {
            $postString .= urlencode($key) . '=' . urlencode($val) . '&';
        }
        $postString = rtrim($postString, '&');
    }

    # PREPARE THE CURL CALL
    $curl = curl_init();
    curl_setopt( $curl, CURLOPT_URL,            $url         );
    curl_setopt( $curl, CURLOPT_HEADER,         FALSE        );
    curl_setopt( $curl, CURLOPT_POST,           TRUE         );
    ($postArray != NULL) ? curl_setopt( $curl, CURLOPT_POSTFIELDS,     $postString ) : '';
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout     );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE         );

    # EXECUTE THE CURL CALL
    $htm = curl_exec($curl);
    $err = curl_errno($curl);
    $inf = curl_getinfo($curl);

    # ON FAILURE
    if (!$htm) {
        # PROCESS ERRORS HERE
        if ($errorReport) {
            echo "CURL FAIL: {$url} TIMEOUT={$timeout}, CURL_ERRNO={$err}";
            echo "<pre>\n";
            var_dump($inf);
            echo "</pre>\n";
            createLog("CURL FAIL: {$url} TIMEOUT={$timeout}, CURL_ERRNO={$err}");
        }
        curl_close($curl);
        return FALSE;
    }

    # ON SUCCESS
    curl_close($curl);
    return $htm;
 }

Problem

I’m using cURL in order to execute some bots. For example;

$url = 'http://siteurl.com/crons/transfer_members.php';
$response = curlPost($url, NULL, 10);
echo ($response) ? "{$url} executed" : "Couldn't execute {$url}";

It is working as expected but I have tons of members so I had to use pagination such as;

http://siteurl.com/crons/transfer_members.php?page=1

http://siteurl.com/crons/transfer_members.php?page=2 etc..

It is just transferring first result set and it is not going for other pages. If I execute it manually it is redirecting to next page (if there are records).

What should I do at this point? I’m quite stuck and I am out of ideas at the moment. I will be glad if anyone could help me out with this problem.

Note:

Everything is in the same domain. No tracing any referrer, not checking any cookie. Just pure PHP pages are called. If there is any pagination in the page which was called by cURL I use meta redirects to redirect to next page.

I don’t want to use cURL inside of a loop. If called php page needs pagination I add following code inside of called php page;

# REDIRECT
echo '<meta http-equiv="refresh" content="' . $timer . ';url=' . $url . '">';

EXTRA INFORMATION

I have crons table in my database and I define $url with looping crons table’s records. Some php pages might have pagination, some don’t have any pagination. That’s why I really would like to avoid using cURL calls inside of the loop to paginate.

If pagination is needed, there is pagination in called PHP page. If I visit the called page on browser than it is redirecting me to the next page. I expected the same thing when I call the page with cURL.

Meaning;
if I visit http://siteurl.com/crons/transfer_members.php on my browser it is redirecting me to next page with meta redirect (example : http://siteurl.com/crons/transfer_members.php?page=x)

I expected the same thing with cURL. When cURL would call http://siteurl.com/crons/transfer_members.php it should go through next pages till all pages are finished. It is just visiting the first page and stopping there.

Answer 1

I just had a different approach.
I returned http://siteurl.com/cron/transfer_members.php?page=n. If returned result starts with http:// or https:// then I called cURLonce again till returned data is finished.

I still didn’t use cURL in the loop (it would cause other unwanted results) but still managed to make it work.

Special thanks to @Shiplu.