Question

My server deployment script triggers a long-running process through SSH, like so:

ssh host 'install.sh'

Since my internet connection at home is not the best, I can sometimes be disconnected while the install.sh is running. (This is easily simulated by closing the terminal window.) I would really like for the install.sh script to keep running in those cases, so that I don’t end up with interrupted apt-get processes and similar nuisances.

The reason why install.sh gets killed seems to be that stdout and stderr are closed when the SSH session is yanked, so writing to them fails. (It’s not an issue of SIGHUP, by the way — using nohup makes no difference.) If I put touch ~/1 && echo this fails && touch ~/2 into install.sh, only ~/1 is created.

So running ssh host 'install.sh &> install.out' solves the problem, but then I lose any “live” progress and error output.

So my question is: What’s an easy/idiomatic way to run a process through SSH so that it doesn’t crash if SSH dies, but so that I can still see the output as it runs?

---

Solutions I have tried:

• When I run things manually, I use screen for cases like this, but I don’t think it will be of much help here because I need to run install.sh automatically from a shell script. Screen seems to be made for interactive use (it complains “Must be connected to a terminal.”).

• Using install.sh 2>&1 | tee install.out didn’t help either (silly of me to think it might).

• You can redirect stdout/stderr into install.out and then tail -f it. The following snippet actually works:

  touch install.out &&  # so tail does not bark (race condition)
  (install.sh < /dev/null &> install.out &
  tail --pid "$!" -F install.out)
  

  But surely there must a less awkward way to do the same thing?

Answer 1

Try using screen:

screen ./install.sh    

If your ssh session gets interrupted, you can simply reattach to the session via another ssh connection:

screen -x

You can provide a terminal to your ssh session using the -t switch:

ssh -t server screen ./install.sh