Question

Okay, this is kinda driving me nuts… I keep receiving an “Internal Server Error” trying to connect to an external FTP connection with Curl. I can access the FTP fine normally, but not through Curl.

I’m using CodeIgniter as a wrapper, but I really don’t think that will cause such an issue. I’ve tried increasing my memory/timeout, but I still can’t get in. The 500 internal server error is actually on my page; I can’t figure out if Curl is returning anything, but I do know that I just get a normal error through Curl (not an internal server error) if I disable the ‘username’ or trying to add a ‘password’ (there is no password for this FTP Login).

Here are my main scripts:

function FTPScrape() {
    parent::Controller();
    $this->load->model("Curl_m");
}

function index() {
    ini_set('memory_limit', '70000000M');
    set_time_limit(0);
    define('COOKIES_DIR', 'cookies');
    define('RANDOM', COOKIES_DIR . '/' . md5(uniqid(mt_rand(), true)));
    $this->Curl_m->init(TRUE, RANDOM . '.txt');
    $this->Curl_m->start();

    $referer = '';
    $url = 'ftp://ftp.server.com/';
    $str = $this->Curl_m->ftp($url, 'user', '', __line__, $referer);
    print "<br><br><textarea cols=\"80\" rows=\"20\">{$str}</textarea><br><br>";
    $this->Curl_m->close();
}

Here are the “Curl_m” model functions I use:

function init($cookies = TRUE, $cookie = 'cookies.txt', $compression = '', $proxy = '') {
if(!is_dir('files')) {
    mkdir('files', 0777);
}
if(!is_dir('cookies')) {
    mkdir('cookies', 0777);
}
else if($dh = opendir('files/')) {
    while(($file = readdir($dh)) !== false) {
        if(is_file('files/'.$file)) {
            unlink('files/'.$file);
        }
    }
}
$this->user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729)';
$this->compression = $compression;
$this->proxy = $proxy;
$this->cookies = $cookies;
$this->filenumber = 0;
$this->follow_location = 1;
$this->headers_html = 0;
$this->binary = 0;
if($this->cookies == TRUE) {
    $this->cookie($cookie);
}
}

function start() {
    $this->process = curl_init();
}

function close() {
    curl_close($this->process);
}

function ftp($url, $user = '', $pass = '', $line_no = '', $referer = '') {
    return $this->execute($url, '', $line_no, $referer, $user, $pass);
}

function execute($url, $data = '', $line_no = '', $referer = '', $user = '', $pass = '') {
    if(isset($this->headers)) {
        unset($this->headers);
    }
    if(preg_match('/\w/xsi', $data)) {
        $type = 'POST';
    }
    else {
        $type = 'GET';
    }

    $host = parse_url($url);
    $this->headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
    $this->headers[] = 'Accept-Language: en-us,en;q=0.5';
    $this->headers[] = 'Accept-Encoding: gzip,deflate';
    $this->headers[] = 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7';
    $this->headers[] = 'Keep-Alive: 300';
    $this->headers[] = 'Connection: keep-alive';
    $this->headers[] = 'Expect:';
    if(preg_match('/\w/xsi',  $referer)) {
        $this->headers[] = 'Referer: ' . $referer . '';
    }

    if($type == 'POST') {
        if(isset($this->Content_type_change)) {
            $this->headers[] = 'Content-Type: ' . $this->Content_type_change . '';
            unset($this->Content_type_change);
        }
        else {
            $this->headers[] = 'Content-Type: application/x-www-form-urlencoded';
        }
        if(isset($this->extra_headings)) {
            $this->headers[] = $this->extra_headings;
        }
        if(!preg_match('/https:/xsi', $url)) {
            $this->headers[] = 'Content-Length: ' . strlen($data) . '';
        }
    }

    curl_setopt($this->process, CURLOPT_URL, $url);
    curl_setopt($this->process, CURLOPT_HTTPHEADER, $this->headers);
    curl_setopt($this->process, CURLOPT_HEADER, $this->headers_html);
    curl_setopt($this->process, CURLOPT_USERAGENT, $this->user_agent);
    if($this->cookies == TRUE) {
        curl_setopt($this->process, CURLOPT_COOKIEFILE, $this->cookie_file);
    }
    if($this->cookies == TRUE) {
        curl_setopt($this->process, CURLOPT_COOKIEJAR, $this->cookie_file);
    }
    curl_setopt($this->process, CURLOPT_ENCODING, $this->compression);
    curl_setopt($this->process, CURLOPT_CONNECTTIMEOUT, 0);
    curl_setopt($this->process, CURLOPT_TIMEOUT, 1200);
    curl_setopt($this->process, CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($this->process, CURLOPT_RETURNTRANSFER, 1);

    if(strpos("ftp", $url) !== false) {
        curl_setopt($this->process, CURLOPT_FTPLISTONLY, 1);
    }
    if(!empty($user)) {
        $usrpwd = $user . ':' . $pass;
        curl_setopt($this->process, CURLOPT_USERPWD, $usrpwd);
    }

    if($this->binary == 1) {
        curl_setopt($this->process, CURLOPT_BINARYTRANSFER, 1);
    }

    if($this->follow_location == 1) {
        curl_setopt($this->process, CURLOPT_FOLLOWLOCATION, 1);
    }
    else {
        curl_setopt($this->process, CURLOPT_FOLLOWLOCATION, 0);
    }

    if($type == 'POST') {
        curl_setopt($this->process, CURLOPT_POSTFIELDS, $data);
        curl_setopt($this->process, CURLOPT_POST, 1);
    }

    if(preg_match("/\w/", $this->proxy)) {
        curl_setopt($this->process, CURLOPT_PROXY, $this->proxy);
    }

    $return = curl_exec($this->process);
    if($this->file_extension($url,$return) == 'jpg') {
        $fh = fopen('captcha.jpg', "w");
        fwrite($fh, $return);
        fclose($fh);
    }
    unset($this->headers);
    return $return;
}

Does anyone know why I may be having this issue?

Most of the script was created before I started this project (namely the functions in the Curl_m model) I just converted the class into an actual codeigniter model.

If I can figure out how to prevent this from causing an internal server error I should be able to fix the rest easily enough.

Answer 1

Well, I figured out why it wasn’t working right. I tried to increase allowed memory and allow unlimited timeout, but I guess my server just didn’t want to allow me to do that.

After I spent yesterday and today trying to configure the Amazon EC2 account my client set up so that I could put all of this information on there, I tested the FTP Scraper again and it successfully connected and retrieved the data I was trying to access. My server was really only for testing, anyway–the final script was going to be put on Amazon EC2, but I was putting it off due to the complexity of setting it up (I’d never used Amazon EC2 before).

Either way, the issue was that the script was either timing out or exceeding the allotted memory. By setting it up on a higher-end server I got the connection to work just fine. Apparently logging into FTP with Curl takes more resources/time than I thought it would.