Question

I have a bash script that processes all of the files in a directory using a loop like

for i in *.txt
do
ops…..
done

There are thousands of files and they are always processed in alphanumerical order because of ‘*.txt’ expansion.

Is there a simple way to random the order and still insure that I process all of the files only once?

Answer 1

Assuming the filenames do not have spaces, just substitute the output of List::Util::shuffle.

for i in `perl -MList::Util=shuffle -e'$,=$";print shuffle<*.txt>'`; do
    ....
done

If filenames do have spaces but don’t have embedded newlines or backslashes, read a line at a time.

perl -MList::Util=shuffle -le'$,=$\;print shuffle<*.txt>' | while read i; do
    ....
done

To be completely safe in Bash, use NUL-terminated strings.

perl -MList::Util=shuffle -0 -le'$,=$\;print shuffle<*.txt>' |
while read -r -d '' i; do
    ....
done

---

Not very efficient, but it is possible to do this in pure Bash if desired. sort -R does something like this, internally.

declare -a a                     # create an integer-indexed associative array
for i in *.txt; do
    j=$RANDOM                    # find an unused slot
    while [[ -n ${a[$j]} ]]; do
        j=$RANDOM
    done
    a[$j]=$i                     # fill that slot
done
for i in "${a[@]}"; do           # iterate in index order (which is random)
    ....
done

Or use a traditional Fisher-Yates shuffle.

a=(*.txt)
for ((i=${#a[*]}; i>1; i--)); do
    j=$[RANDOM%i]
    tmp=${a[$j]}
    a[$j]=${a[$[i-1]]}
    a[$[i-1]]=$tmp
done
for i in "${a[@]}"; do
    ....
done