Question

I am trying to pass credentials to a website so I can use file_get_contents on it to extract some data but it is not working, I am getting a blank page so any idea what is wrong here?

<?php


$username="[e-mail]";
$password="Koin";

$url="confluence.rogersdigitalmedia.com";


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);

$str= file_get_contents("confluence.rogersdigitalmedia.com/display/prodsupport/Team+Calendar");
echo $str;
?>

Here is the new code it is still not working stuck at login screen when I do get contents….

<?php
$username="[e-mail]";
$password="Koin";

$url="confluence.rogersdigitalmedia.com";


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);


//Replaced due to special chars in url for username and pass
//curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
curl_setopt($ch, CURLOPT_USERPWD, urlencode($username) . ':' . urlencode($password));
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);

echo file_get_contents('http://confluence.rogersdigitalmedia.com/exportword?pageId=1114407');
?>

New code: I know $url is the URL for which I have to login, but what do I put in $data? I know it’s my login info, but how do I put it (e.g., <username> space <password>)?

<?php
function do_post_request($url, $data, $optional_headers = null)
{
  $params = array('http' => array(
              'method' => 'POST',
              'content' => $data
            ));
  if ($optional_headers !== null) {
    $params['http']['header'] = $optional_headers;
  }
  $ctx = stream_context_create($params);
  $fp = @fopen($url, 'rb', false, $ctx);
  if (!$fp) {
    throw new Exception("Problem with $url, $php_errormsg");
  }
  $response = @stream_get_contents($fp);
  if ($response === false) {
    throw new Exception("Problem reading data from $url, $php_errormsg");
  }
  return $response;
}

Answer 1

You probably need to escape the @ in the username:

curl_setopt($ch, CURLOPT_USERPWD, urlencode($username) . ':' . urlencode($password));

This is because some characters (such as @ and :) have special meanings in URLs. You need to escape them so that the server will treat them as the characters, rather than as indicating something about the HTTP request.

Scrap all that. Your edit shows that you have a fundamental misapprehension about how HTTP works. You can’t use CURL to pass HTTP login credentials to a site that expects a web form to be filled out and then expect to be able to log in with file_get_contents (which is part of a completely separate API).

In terms of how you actually do this… First, find out whether the site has an API. If it does not, find out whether they actually want you to be screen-scraping. It would be very bad form to screen-scrape a website that doesn’t allow it. It may well in fact be a breach of copyright law.

Second, program to the API. This means logging in in the way that the site enforces. This will be with form values (which you will need to send in CURLOPT_POSTFIELDS, probably) and cookies (which you will need to use in every request, probably using CURLOPT_COOKIEFILE).

Work at understanding how the client-server relationship works, study the API and don’t just chuck code at the problem and expect it to work.