Question

I am trying to figure out if it is possible to set up a cron job that will transfer all files from one folder on my server to another folder and then take a set number of files (files chosen randomly) from another folder and put it in the original folder. If so any hints on how to do this, I have no experience with cron at all, I just don’t want to have to log in with ftp and do the transfers manually.

Answer 1

Cron is really simple, all it’s doing is to run a command of your choice at the specified times of day.

In your case, you probably want to write a shell script that use rsync, scp or ftp to transfer the files, make sure that exits successfully (check exit code from transfer, stored in the $? variable), then move the set of files into the original folder.

I would use rsync and passwordless authentication via ssh keys. That’s good for security, and if you want to, you can even limit the receiving side to only allow that ssh key to run rsync’s server side.

If this script is called /opt/scripts/myscript.sh, and is to be run once every 10 minutes, add the following to your crontab (run crontab -e to edit your crontab):

*/10 * * * * /opt/scripts/myscript.sh

Remember that the environment variables you have available in your shell are not the same as those available when the cronjob runs, so PATH, etc, may be different. This often causes cron jobs to fail the first few times you run them (See my law on cron jobs: http://efod.se/blog/archive/2010/02/19/forsbergs-law-on-cron-jobs :-)). Any output from cron is sent via mail to the user running the cron job, which is helpful for debugging. Simple debugging writing messages to some file in /tmp/ is also often a way to get your cron jobs running.

In many cases it makes sense to run cron jobs as a special user. Don’t run your cron jobs as root unless they absolutely must have root access, it’s better to run things as special users that only has limited permissions in the file system.