Question

I’m writing a little script, that will create archives in main thread and after each archive is complete, a new thread would be created by calling function that would take care of uploading these archives. The reason I want uploading to be done in background is so that another archive could be created while the previous archives are being uploaded.

The problem I’m having is at the very end of the script. That is, main thread don’t wait for all uploading threads to finish before exiting. Look at the following simplified script (I removed/changed parts of the code not related to the problem)

function func {
for files in /home/somewhere/
  do
    echo "Uploading $1" &
  done
wait
}

find /home/some/path -type f | while read filename ; do
  echo "Creating archive of $filename"
  func $somevariable &
done

wait

Everything is executing very nicely until the last archive is created, then the script ends before all func threads finish, leaving many files not uploaded.

Thank you for your ideas.

Answer 1

Update: good points in the comment.

So, on a second look, it turns out the problem is the subshell that is created by the pipe to the loop. It’s a good way to structure the script but you need to do the final wait in the shell that spun off the background tasks.

So do something like this:

find /home/some/path -type f | (while read filename; do
    echo "Creating archive of $filename"
    func $somevariable &
  done
  wait
)