Question

I have a script that runs in AIX ksh that looks like this:

wc dir1/* dir2/* | {awk command to rearrange output} | {grep command to filter more} > dir2/output.txt

It is a precondition to this line that dir2/output.txt does not exist.

The issue is that dir2/output.txt has contained itself in the output (it’s happened a handful of times out of hundreds of times with no problem). dir1 and dir2 are NFS-mounted.

Is it related to the implementation of wc — what if the the first parameter takes a long time? I think not, as I’ve tried the following:

wc `sleep 5` *.txt > out.txt

Even in this case out.txt does not list itself.

As a last note, wildcards are used in this example where they are used in the actual script. So if the expansion happens first, why does this problem occur?

At what point is dir2/output.txt actually created?

Answer 1

Redirections are done by the shell, as are globs. Your problem is that, in the case of a pipeline, each pipeline stage is a separate subprocess; whether the shell subprocess that does the final redirection runs before the one that builds the glob of input files for wc will depend on details of the scheduler and system load, among other things, and should be considered indeterminate.

In short, you should assume that this will happen and either exclude dir2/output.txt (take a look at ksh extended glob patterns; in particular, something along the lines of dir2/!(output.txt) may be useful) or create the output somewhere else and mv it to its final location afterward.