Question

I have a static page, which contains multiple links to text files on the server. (around 100 links which all open a text file contents on the web browser itself).

The text files have text and numbers, where I need to get a specific number with a percentage (i.e., 52.56%), and store it to another text/csv file with the name of the link.

Links are basically in a single line format:

VAA007
VAB038
VAC041
VAD050
VAE031
VEA032
VEB053
VEC044
VEF015

The text files has the following values, along with lots of other text and numbers:

# Result Summary:
# Overall Run:191,Not Run:161,Covered:54.26%

The link address/URL is in the following format:

https://myWeb.local/~gags/cgi-bin/latestRun.pl
https://myWeb.local/~gags/cgi-bin/showReport.pl?fn=/OUT/VAA007.txt       
https://myWeb.local/~gags/cgi-bin/showReport.pl?fn=/OUT/VAD050.txt

Is there some easy way to work out the above and get the results into a text file or CSV file?

Answer 1

This should do the trick:

!/bin/sh
URL=https://myWeb.local/~gags/cgi-bin/showReport.pl?fn=/OUT/
for file in `cat links.txt`
do
  curl -s "$URL$file.txt" | awk -F':' '/^# Overall/ {print $NF}' > "$file.txt"
done

Assuming the links are stored in the file links.txt. If you need to fetch them first, you could add curl -s https://someurl > links.txt to the top of the script.

You don’t specify if you have to log on to the server, and if so what kind of log on scheme is used. If that’s a requirement, that’s easily handled with curl.