Question

I want a script which automates the running of big mathematical computing jobs on a remote machine. Currently I do this:

• ssh into remote machine; run matlab script
• periodically check in to see if job is done yet
• when job is done, I manually scp the files containing the results back to my home machine.

I tried to use these kind of lines in my script to do this job (note the script runs on my machine):

ssh nohup matlab -r theScript; exit;
scp remote@~/files ~/files

This doesn’t work. After a while the ssh session ends and the script just proceeds to do the scp, even though the job is not finished yet and the files don’t exist yet.

I think what I need to do is check periodically whether the job is done, perhaps by ssh’ing in periodically and reading the nohup.out file looking for a DONE! signal using grep. Then when I see that, copy the files back. But this seems complicated and I don’t know how to get the DONE! message back to my computer to run a conditional on (if you see the DONE signal, do this stuff…) any ideas?

Answer 1

Yeah, the ssh could time out or something. So yeah, you best option is to poll. e.g.

RUN="$(date +%Y%m%d-%H%M%S)"
ssh remote " nohup bash -c \"( matlab -r theScript; echo \$? > $RUN.done ) >$RUN.log 2>&1 </dev/null &\" "
DONE=""
while [ -z "$DONE" ]
do
    sleep 60
    DONE="$(ssh cat $RUN.done 2>/dev/null)"
done
if [ $DONE -eq 0 ]
then
    scp ...
else
    # Optionally fetch logfile
    # scp $RUN.log@remote ...
    echo "ERROR in remote matlab...."
fi