Question

I’m trying to write a bash script that will let me download multiple web pages using curl. For each webpage, I want to be able to pass curl the page and the referer link. I want to be able to supply multiple webpages at once.

In other words, I want to be able to loop through the webpages I supply the script, and for each page, pass the associated webpage and referer link to curl.

I thought I’d use an array to store the webpage and referer link in a single variable, thinking that I could then extract the individual elements of the array when running curl.

My problem is that I can’t figure out how to get multiple arrays to work properly in a for loop. Here is an idea of what I want to do. This code does not work, since “$i” (in the for loop) doesn’t become an array.

#every array has the information for a separate webpage
array=( "webpage" "referer" )
array2=( "another webpage" "another referer" )

for i in "${array[@]}" "${array2[@]}" #line up multiple web pages
do
    #use curl to download the page, giving the referer ("-e")
    curl -O -e "${i[1]}" "${i[0]}"
done

If I was only working with one array, I could easily do it like this:

array=( "webpage" "referer" )
REFERER="${array[1]}"
PAGE="${array[0]}"
#use curl to download the page, giving the referer ("-e")
curl -O -e "$REFERER" "$LINK"

It’s once I have more than one webpage that I want to process at once that I can’t figure out how to do it correctly.

If there is another way to handle multiple webpages, without having to use arrays and a for loop, please let me know.

Answer 1

Thanks to everyone for their responses. Both ideas had merit, but I found some code in the Advanced Bash Guide that does exactly what I want to do.

I can’t say I fully understand it, but by using an indirect reference to the array, I can use multiple arrays in the for loop. I’m not sure what the local command does, but it is the key (I think it runs a sort of eval and assigns the string to the variable).

The advantage of this is that I can group each webpage and referer into their own array. I can then easily add a new website, by creating a new array and adding it to the for loop. Also, should I need to add more variables to the curl command (such as a cookie), I can easily expand the array.

function get_page () {
        OLD_IFS="$IFS"
        IFS=$'\n'       #  If the element has spaces, when using
                        #  local to assign variables

        local ${!1}


        # Print variable
        echo First Variable: "\"$a\""
        echo Second Variable: "\"$b\""
        echo ---------------
        echo curl -O -e "\"$a\"" "\"$b\""
        echo  
        IFS="$OLD_IFS"
}       

#notice the addition of "a=" and "b="
#this is not an associative array, that would be [a]= and [b]=
array=( a="webpage" b="referer" )
array2=( a="another webpage" b="another referer" )

#This is just a regular string in the for loop, it doesn't mean anything
#until the indirect referencing later
for i in "array[*]" "array2[*]" #line up multiple web pages
do
        #must use a function so that the local command works
        #but I'm sure there's a way to do the same thing without using local
        get_page "$i" 
done

This results in:

First Variable: "webpage"
Second Variable: "referer"
---------------
curl -O -e "webpage" "referer"

First Variable: "another webpage"
Second Variable: "another referer"
---------------
curl -O -e "another webpage" "another referer"