Question

I am trying to write a simple bash script that will use a list from a text document and curl each URL that is on the list in order to see what the contents of each URL is. It allows me to cURL 2 sites and creates the text documents for the rest however it only downloads the first 2. I have already manage to write the script that pulls there IP’s and places them in a seperate file using the grep command. At first i tried

#!/bin/bash
for var in `cat host.txt`; do
curl -s $var >> /tmp/ping/html/$var.html
done

I have tried with and without the silent switch. I then tried the following:

#!/bin/bash  
for var in `head -2 host.txt`; do  
curl $var >> /tmp/ping/html/$var.html  
wait  
done  
for var in `head -4 host.txt | tail -2`; do  
curl $var >> /tmp/ping/html/$var.html  
done  

This would try and do them all at the same time again stopping after 2

#!/bin/bash  
for var in `head -2 host.txt`; do  
curl $var >> /tmp/ping/html/$var.html   
done 
wait 
for var in `head -4 host.txt | tail -2`; do  
curl $var >> /tmp/ping/html/$var.html  
done  

This would do the same, I am new to bash scripting and only know some of the basics, any help would be appreciated

Answer 1

Start with the simple: verify that you are in fact iterating over the entire list:

# This is the recommended way to iterate over the file. See
# http://mywiki.wooledge.org/BashFAQ/001
while read -r var; do
    echo "$var"
done < hosts.txt

Then add in the call to curl, checking its exit status

while read -r var; do
    echo "$var"
    curl "$var" >> /tmp/ping/html/$var.html || echo "curl failed: $?"
done < hosts.txt