Question

I’m writing a shell script, and in an effort to make it shorter and easier to read, I’m trying to use nested subshells to pass parameters to diff.

Here’s what I have:

if
  diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' > /dev/null;
then  
  echo There is no difference between the files. > ./participants-by-state-results.txt;
else  
  diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' > ./participants-by-state-results.txt;
fi

When I run the script, I keep getting diff: extra operand 'AL'

I’d appreciate any insight into why this is failing. I think I’m pretty close. Thanks!

Answer 1

Your code is unreadable because the lines are so long:

if diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' \
       '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' \
       > /dev/null;
then  
    echo There is no difference between the files. > ./participants-by-state-results.txt;
else  
   diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' \
      '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' \
      > ./participants-by-state-results.txt;
fi

Repeating whole commands like that is also fairly nasty. You also have major problems with your use of single quotes; you only have one sort in each set of commands, apparently operating on the combined outputs of two identical awk commands (whereas you probably need two separate sorts, one for the output of each awk command); you’re not using the -F option to awk when you could; you are repeating the gargantuan file names all over the place; and finally, it appears that you are probably wanting to use process substitution, but not actually doing so.

Let’s take a step back and formulate the question clearly.

• Given two files (new-participants-by-state.csv and current-participants-by-state.csv) find the first pipe-separated field on each line of each file, sort the lists of those fields, and compare the results of the two sorted lists.
• If there are no differences, write a message into the output file participants-by-state-results.txt; otherwise, list the differences in the output file.

So, we could use:

oldfile='current-participants-by-state.csv'
newfile='new-participants-by-state.csv'
outfile='participants-by-state-results.txt'

tmpfile=${TMPDIR:-/tmp}/xx.$$

awk -F'|' '{print $1}' $oldfile | sort > $tmpfile.1
awk -F'|' '{print $1}' $newfile | sort > $tmpfile.2

if diff -iy $tmpfile.1 $tmpfile.2 > $outfile
then echo "There is no difference between the files" > $outfile
fi

rm -f $tmpfile.?

If this was going to be the final script, we’d want to put trap handling in place so that the temporary files are not left around unless the script is killed dead with SIGKILL.

However, we can now use process substitution to avoid the temporary files:

oldfile='current-participants-by-state.csv'
newfile='new-participants-by-state.csv'
outfile='participants-by-state-results.txt'

if diff -iy <(awk -F'|' '{print $1}' $oldfile | sort) \
            <(awk -F'|' '{print $1}' $newfile | sort) > $outfile
then echo "There is no difference between the files" > $outfile
fi

Note how the code carefully preserves symmetries where there are symmetries. Note the use of shortish variable names to avoid the repetition of long file names. Note that the diff command is run just once, not twice – throwing away results which are needed later is not very sensible.

You could compress the output I/O redirection even more using:

{
if diff -iy <(awk -F'|' '{print $1}' $oldfile | sort) \
            <(awk -F'|' '{print $1}' $newfile | sort)
then echo "There is no difference between the files"
fi
} > $outfile

That sends the standard output of the enclosed commands to the file.

Of course, CSV might not be the appropriate nomenclature if the files are pipe-separated rather than comma-separated, but that’s another matter altogether.

I’m also assuming that the status from diff -iy works as suggested by the original script; I’ve not validated that usage of the diff command.