Question

Im trying to grab all data(text) coming from a URL which is constantly sending text, I tried using PHP but that would mean having the script running the whole time which it isn’t really made for (I think). So I ended up using a BASH script.

At the moment I use wget (I couldn’t get CURL to output the text to a file)

wget --tries=0 --retry-connrefused http://URL/ --output-document=./output.txt

So wget seems to be working pretty well, apart from one thing, every time I re-start the script wget will clear the output.txt file and start filling it again, which isn’t what I want. Is there a way to tell wget to append to the txt file?

Also, is this the best way to capture the live stream of data?
Should I use a different language like Python or …?

Answer 1

You can do wget --tries=0 --retry-connrefused $URL -O - >> output.txt.

Explanation: the parameters -O is short for --output-document, and a dash - means standard output.

The line command > file means write “write output of command to file“, and command >> file means “append output of command to file” which is what you want.