Question

I have a bash file that contains wget commands to download over 100,000 files totaling around 20gb of data.

The bash file looks something like:

wget http://something.com/path/to/file.data

wget http://something.com/path/to/file2.data

wget http://something.com/path/to/file3.data

wget http://something.com/path/to/file4.data

And there are exactly 114,770 rows of this. How reliable would it be to ssh into a server I have an account on and run this? Would my ssh session time out eventually? would I have to be ssh’ed in the entire time? What if my local computer crashed/got shut down?

Also, does anyone know how many resources this would take? Am I crazy to want to do this on a shared server?

I know this is a weird question, just wondering if anyone has any ideas. Thanks!

Answer 1

Use

#nohup ./scriptname &>logname.log

This will ensure

• The process will continue even if ssh session is interrupted
• You can monitor it, as it is in action

Will also recommend, that you can have some prompt at regular intervals, will be good for log analysis. e.g. #echo '1000 files copied'

---

As far as resource utilisation is concerned, it entirely depends on the system and majorly on network characteristics. Theoretically you can callculate the time with just Data Size & Bandwidth. But in real life, delays, latencies, and data-losses come into picture.

So make some assuptions, do some mathematics and you’ll get the answer 🙂