Question

In my bash script I have the following (for concreteness I preserve the original names;
sometimes people ask about the background etc., and then the original names make more sense):

 tail -n +2 Data | while read count phi npa; do
   cat Instances/$phi >> $nF
 done

That is, the first line of file Data is skipped, and then all lines, which are of
the form “r c p n”, are read, and the content of files Instances/p is appended
to file $nF (in the order given by Data).

In typical examples, Data has millions of lines. So perhaps I should write a
C++ application for that. However I wondered whether somebody knew a faster
solution just using bash?

Answer 1

Here I use cut instead of your while loop, but you could re-introduce that if it provides some utility to you. The loop would have to output the phy variable once per iteration.

tail -n +2 Data | cut -d' ' -f 2 | xargs -I{} cat Instances/{} >> $nF

This reduces the number of cat invocations to as few as possible, which should improve efficiency. I also believe that using cut here will improve things further.