Question

set +f; rm *; touch a; for i in *; do touch b; echo $i; done

In all the shells I tried (dash, ksh, zsh, bash), the above snippet outputs only “a”. Implementing the same in C, (opendir/loop on readdir creating a file) also outputs only “a”. However, if the directory contains enough files (~4096), the C implementation usually outputs “b” as well. (ie, readdir returns a result for the file created after opendir). I see nothing in the shell standard that indicates how the shell ought to respond in this case. Could a standard compliant shell enter the loop for the file created after the glob? This would be a highly desirable trait, since it would mean the shell is not reading the entire glob into memory before performing any actions. In situations where directories are expected to contain many files, it often takes several seconds just to read the glob into memory, which is time wasted.

Is there any shell implementation that does not read the entire glob into memory before entering the loop?

Answer 1

No. The context into which the glob is expanding is essentially identical to an ordinary command expansion context wherein all expansions are processed and the resulting words saved in an immutable fashion for iteration. There is no lazy iterator possible with a for-in loop. Of course, expansions can be side-effectful and mixed with globs, so they have to be eagerly evaluated. This is why find -exec [+;] is still so frequently recommended over globstar when it’s possible to do things concurrently.

I can’t really say anything about this 4096 issue. I don’t think the two are really comparable. Shell for..in just expands the words and iterates them.

A related FAQ is whether you can do things like read ahead to the next values to be assigned. As far as I’m aware there are no bourne-like shells that give any extra access to the word list. You’d have to use arrays for that. Basically all the limitations of for..in can be overcome via arrays.

Here is a funny lazy coproc generator I wrote for Bash. It’s pretty useless.

coproc x { while :; do find . -type f -maxdepth 1 -exec sh -c 'read; echo "$1"' -- {} \;; done; };

while :; do
    echo 1 >&"${x[1]}"
    read -ru "${x[0]}" file
    echo "$file"
    sleep 1
done

And one more for..in tidbit that doesn’t really have to do with the question — In ksh93, and Bash’s git devel branch, it is possible to make use of the “control variable” in an interesting way.

function f {
    nameref x # Chet may decide not to emulate the typeset -n aliases

    for x; do
        x=hi
    done
}

typeset -a arr
f 'arr['{0..3}']'
typeset -p arr # arr=(hi hi hi hi)

Each iteration assigns a reference to the given object to x. Of course in ksh that can be any arbitrary complex datatype. I suppose this might be abused to somehow simulate lazyness. Unfortunately this pattern doesn’t seem to work in mksh.

Edit Forgot since writing this that I found a lot of shells do in fact optimize for x syntax. I assume at least for x in is copy-on-write and only makes a copy of the positional parameters if shift or set is used within the loop.