Question

I am migrating a legacy bash script which runs inside a www server. I have came across an issue that I am not able to explain.

When ran through the webserver (as a CGI included script), the following commands:

#!/bin/bash
set -e -u
echo "BP: Statistics generation 1" >/dev/fd/2
echo "BP: one: $1, two: $2" >/dev/fd/2

more ../download/interactions.$1.$2 | grep -v "#" | awk '($2!~/_R/){print $2}' | sort -k 1 | uniq > rna.names.txt
more ../download/interactions.$1.$2 | grep -v "#" | awk '($1!~/_R/){print $1}' | sort -k 1 | uniq > protein.names.txt

The protein.names.txt looks like:

../download/interactions.3424.asdfjj
507-558_1
::::::::::::::

The middle is the expected output but the first and the last line not. Neither of them are contained in the interactions.x.y file, which only contains a title in the beginning describing the columns and data. Test grep shows confirms that the data is not present.

The weirdest thing is that if I log in to the same machine and run the script from terminal (even su-ing as the www-data), the output is just the middle line, as it should be.

I have tried to export env and set from bash right before the said command and re-import them before running the script trying to recreate what is happening on the server, with no avail.

The script itself is not invoked directly, there is a CGI-included perl-script executing a system call to run the script via bash.

What is the source of that output? How can I at least reproduce it?

Answer 1

There is no need to use more, and it is likely the source of the ‘:::’s you’re getting.
more is a paging facility, and it expects a key press (maybe the space bar), to tell it, ‘display the next page’. In this context your script isn’t giving any input to keep the data moving. I’m guessing here but the ‘:’ is a prompt char.

The super-short cut is to give grep the filename, i.e.

grep -v "#" ../download/interactions.$1.$2 | awk '($2!~/_R/){print $2}' | sort -k 1 | uniq > rna.names.txt

Or if you have a manager that is suspicious of change, then

cat ../download/interactions.$1.$2 | grep -v "#" | awk '($2!~/_R/){print $2}' | sort -k 1 | uniq > rna.names.txt

Finally, the most efficent solution would be

   awk '($0 !~ /#/ && $2 !~ /_R/){print $2}' ../download/interactions.$1.$2 | sort -u -k > rna.names.txt

(I don’t think you need the ( ) surrounding your $2!~/_R/ test.)

I hope this helps.