Question

Having a large set of files I need to grep through, I’m doing it within a bash script like this:

#! /usr/bin/env bash

REX="word"
grep -IP $REX A* >> result &
grep -IP $REX B* >> result &
grep -IP $REX C* >> result &
grep -IP $REX D* >> result &
[..]

How to know when all the processes are finished?

Answer 1

I would go about solving your problem a different way. find can find specific files on your filesystem, and xargs allows you to run commands on given input files. So I would use a command like this:

find ./ -type f -print0 | xargs -0 -I{} -n1 -P4 grep -IP $REX '{}' >> result

This would search, starting in the current directory (./) for all regular files (-type f), and pass them along to xargs safely in case there are any spaces in the filename (-print0). xargs then, for each command, runs your grep command. -I{} tells xargs that where it sees {} it will insert the filename into the command. Not strictly necessary here, but good practice. -0 goes hand in hand with -print0 from find, and tells it to expect input that way. -P4 tells xargs to run up to 4 processes at the same time, and -n1, as described by the man page, hints to xargs to only use one argument at a time per command.

There are various tweaks you can make here, whether it’s not wanting to search for all files, or to only go to a certian depth, but this general command should get you started with this kind of task.