Question

I need to run the following command in a folder containing a lot of gzipped files.

perl myscript.pl -d <path> -f <protocol> "<startdate time>"
 "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>

My problem is that the command does not take gzipped files as input. So, I would need to unzip all those gzipped files on the fly and run this command on those unzipped files. These gzipped files are in another folder where I am not allowed to unzip them. I want a shell script that will take the path of the remote gzipped files and store it under path (which is also going to be a argument to the script). Do the unzipping and then run the above command.

N.B: The “protocol”, “startdate time”, “enddate time”, “outputfilename” don’t have to be arguments for now I will just put them directly in the script so that it is less complex.

Answer 1

You can do:

for fname in path/to/*.gz; do gunzip -c "$fname" | perl myscript.pl ; done

Expanded:

for fname in path/to/*.gz; do
  gunzip -c "$fname" | perl myscript.pl
done

And to make it accept filenames with spaces:

old_IFS=$IFS
IFS=$'\n'
for fname in path/to/*.gz; do
  gunzip -c "$fname" | perl myscript.pl -f <protocol> "<startdate time>" \
     "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>
done
IFS=$old_IFS

This way, you make the script read standard input, which will contain the file content, without having to use temporary files.

---

EDIT: Here’s a wrapper script that solves the problem like initially suggested in the question:

`myscriptwrapper`:
#!/bin/bash
gzip_path="$1"
temp_path="$2"
#loop thru files from gzip_pah\th
for fname in $gzip_path/*.gz; do
  basename=`basename $fname`
  #fill them in the target dir
  gunzip "$fname" -c > "$temp_path/$basename"
done

#finally, call our script
perl myscript.pl -d "$temp_path" -f <protocol> "<startdate time>" "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>

---

EDIT 2: Using tar.gz files:

`myscriptwrapper`:
#!/bin/bash
gzip_path="$1"
temp_path="$2"
cd "$temp_path"
#loop thru files from gzip_pah\th
for fname in $gzip_path/*.tar.gz; do
    tar -xzf $fname
done

#finally, call our script
perl myscript.pl -d "$temp_path" -f <protocol> "<startdate time>" "<enddate time>" -o "0:00 23:59" -v -g -b <outputfilename>