Question

I wrote this script to automate wgets from my different servers, between each other, and to output the results to a log file, and email them to me. Part of the reason I am sharing this is so that other people can use the solution for themselves.

I’ve placed the file in cron.weekly so it sends me weekly emails.

Btw. I am an absolute novice, this was the first script I wrote by learning from this site.

Script is as follows:
I have done some minor commenting from my side as well.

#!/bin/sh
#wget -verbose -tries -timeout -dontcreatedirectories -dontsaveanything  http://x  output text to file

# automating apt-get update and upgrade so it runs weekly
apt-get update
apt-get upgrade -y

# this is where the logs will be stored
cd /var/log

# create a tmp file and if it is left behind from earlier results, empty it out
echo -e "\n" > wget.tmp

# these are the header texts for the results 
echo "***************" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
echo "WGET SCHEDULES" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
echo "***************" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log

# this will make note of the time the tests are started
date "+START OF WGETS SCHEDULE at %c"  >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
echo "\n" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
echo "\n" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log

# up to 10 servers to wget from
echo "Server #1" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
wget -v -r -t 2 -T 7 -nd -O /dev/null http://servername/file 2>> wget.tmp
grep -i 'Length\|connect\|100%\|saved\|failed\|Error' wget.tmp >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
rm -f wget.tmp
echo "\n" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log

.
continues till
.

echo "Server #10" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
wget -v -r -t 2 -T 7 -nd -O /dev/null http://servername/file 2>> wget.tmp
grep -i 'Length\|connect\|100%\|saved\|failed\|Error' wget.tmp >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
rm -f wget.tmp
echo "\n" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log

# record the end time of tests
echo "***************" >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log
date "+END OF WGETS SCHEDULE at %c"  >> $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log

#mail results to yourself
echo "Subject: $(hostname -s| tr a-z A-Z)"  Wget Results v.2.0 Sendmail  "$(date +%x)"      ----   "$(date "+%A")"    "$(date "+%X")   ("$(date "+%Z")")  :::  Week "$(date "+%V")"  """ | cat -  $(hostname -s)"-wget-SUMMARY-"$(date "+%b%d_%Y" | tr A-Z a-z).log | /usr/lib/sendmail -t myemailaddress@host.x

ideally, i would like to not limit myself to x # of servers, but so long as there is text in file servers.list (by included ‘.’ in the filename i can store it in cron.weekly and it will not be processed). So for each server, write server #, and process the wget, output to file, grep to log file. It would be best if the servers.list file was formatted

hostname filename store/to
.
.
hostname filename store/to

Example:

msn.robots.ua 100mb.test /dev/null
justme.cov ratings.xlsx  /srv/storage/latest

the mails look like

Subject: Servername Wget Results v.2.0 Sendmail 09/09/12 —- Sunday 04:59:21 (MSK) ::: Week 36

---

WGET SCHEDULES

---

START OF WGETS SCHEDULE at Sun Sep 9 04:52:00 2012

Server #1
Connecting to xxx.xxxxxx.com|ip.ip.ip.ip|:80… connected.
Length: 104857600 (100M) [text/plain]
102350K ………. ………. ………. ………. ……….100% 5.73M 0s
102400K 100% 0.00 =25s
2012-09-09 04:52:27 (4.07 MB/s) – `/dev/null’ saved [104857600/104857600]
.
.

---

END OF WGETS SCHEDULE at Sun Sep 9 04:59:21 2012

---

So my /var/log ends up storing the files such as wordpress-wget-SUMMARY-sep24_2012.log as an example file name. Problem is maintaining it is becoming a problem, servers change, so I would like to just update the servers list in one location and let them deal with it all themselves. Prior to execution it would download the new servers.list and the rest is done.

Mail is working output to log file is working, it is all working, just needs a tiny bit of tweaking.

Ty ! Ty ! Ty for reading .

Answer 1

If I’m duplicating your question right, you want to replace out the fact that you can only do 10 servers and want to pull the data from a file instead of having to fix it in the script if anything changes. Based on that duplication, here is what I suggest:

In your server.list file, | delimit it, I’ll explain why after the code snippit. So your server.list file would look like:

msn.robots.ua|100mb.test|/dev/null
justme.cov|ratings.xlsx|/srv/storage/latest

With this set up, you can put a for loop in your bash script with your duplicative code in the middle like this:

    for conf in $(cat server.list); do
        server=`echo $conf | cut -d"|" -f1;`
        filename=`echo $conf | cut -d"|" -f2;`
        storeto=`echo $conf | cut -d"|" -f3;`

        ... the rest of your code goes here using the above variables ...

    done

If you use spaces as the delimiter in your file then $conf in each iteration will be only each space separated value, instead of getting the line and then splitting it up.

There is probably a more elegant way to do this with awk, but this would work.