Question

I have a cron job setup that uses a script to download many files from an FTP to a local server at a set time. It works absolutely fine when I manually run the script, It even works completely fine 9 out of 10 times via the cron. However, occasionally 1 of the files it’s set to get fails, and it throws a similar error to this:

ftp_get(): Opening BINARY mode data connection for Media_4.zip (421623342 bytes). in /home/crons/feeds/get_files.php on line 102

line 102 is the ftp_get(), I know there’s nothing wrong with it as it works fine when run manually/other automated attempts, but for full clarity:

foreach ($download_files as $download_file)
{
  $local_file = "{$path}/files/{$download_file}";
  $download = ftp_get($conn_id, $local_file, $download_file, FTP_BINARY);
  if (!$download) 
  {
    ftp_dl_log($queue_number, "FTP download has failed for $source_name: $download_file", "warn");
  } 
  else 
  {
    ftp_dl_log($queue_number, "Downloaded file successfully");
  }
}

I doubt it’s anything to do with file size, permissions, or anything like that, in total there is ~2gb spread across multiple files in total downloaded, sometimes it seems to just fail on one of the files, all other files are retrieved without a problem so I’m pretty confused. An example of all files would be:

Media_1.zip (600mb, downloads fine)
Media_2.zip (200mb, downloads fine)
Media_3.zip (700mb, downloads fine)
Media_4.zip (400mb, downloads fails)
Media_5.zip (300mb, downloads fine)
Media_6.zip (250mb, downloads fine)
Media_summary.csv (2mb, downloads fine)

Size restriction isn’t an issue, neither is a timeout occuring.

The file I’ve used as an example was uploaded to the ftp @

Oct 31 17:27

and the ftp_get() for the file occured @

Nov 01 03:14

So it couldn’t be that the file was incomplete at the time of request. And considering I can ftp_get() the same file without error later on it isn’t corrupt.

Does anyone know of any other things that could be causing this problem? Could the server performing other fetchs at the same time cause an issue?

Any suggestions or insight would be welcome.

Answer 1

I think you are probably looking at a server issue. If you are running feeds like this simultaneously it could be a load issue.

I would try monitor the server load (using top or htop) at that time and probably re-distribute some other processes to give this one a bit more breathing space.