Question

I’m trying to set up attributes to wget at the start of the program to use them for all the wget instances.

WGET_LOG_FILE=path/to/wget_log
USER_AGENT='Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0x'

WGET="wget -U $USER_AGENT -a $WGET_LOG_FILE --no-clobber --wait=2 --random-wait"

$WGET www.webpage.com

The problem is that it ignores the log and directs everything to the output or gives an error of “path/to/wget_log doesn’t exist”, therefore I suppose it also ignores the User agent string…

So what am I doing wrong? I’ve tried changing ” to ‘ or $() …

The weird thing is that if I execute in the terminal:

wget -U $USER_AGENT -a $WGET_LOG_FILE --no-clobber --wait=2 --random-wait www.webpage.com

It works, so I’m guessing that it’s executed in some subshell and that’s why it doesn’t get it.

So the questions are:

1. How to correctly set attributes to wget in a bash script? Answered by Chepner
2. How to check if wget is using the user agent specified? Just wget any user agent check webpage such as: http://whatsmyuseragent.com/
3. Is 2 seconds a good wait rate to fetch webpages?

Thanks

Answer 1

Generally, you shouldn’t rely on expanding a parameter into the name of the command to run and its arguments; it works as expected for simple cases, but you can run into trouble quickly. I can’t exactly explain the errors you are seeing (since you aren’t quoting $USER_AGENT, I’m surprised even your third example works).

Since you are using bash, you should use an array to hold your arguments, then call wget explicitly and expand the array to provide the arguments.

WGET_LOG_FILE=path/to/wget_log
USER_AGENT='Mozilla/5.0 (X11; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0x'

wget_args=( "-U" "$USER_AGENT"
            "-a" "$WGET_LOG_FILE"
            "--no-clobber"
            "--wait=2"
            "--random-wait")

wget "${wget_args[@]}" www.webpage.com