Question

If I run this command it works fine in the terminal:

for dirname in $(ls -d dir/checkpoint/features.txt/20*);do;echo "hello";done

But when run through /bin/sh -c it gives an error

/bin/sh -c "for dirname in $(ls -d dir/checkpoint/features.txt/20*);do;echo "hello";done"

ERROR:

/bin/sh: -c: line 1: syntax error near unexpected token `dir/checkpoint/features.txt/201108000'
/bin/sh: -c: line 1: `dir/checkpoint/features.txt/201108000'

My default shell is /bin/bash. I cant seem to understand what is causing this. My default implementation for running all shell commands in my program is by appending /bin/sh -c to them. It is the first time i am seeing this issue. Any suggestions?

Answer 1

Don’t try to parse the output of ls, especially with a for construct. There are many, many ways that this can go wrong.

This is a good place to use find instead. Try this:

/bin/sh -c "find dir/checkpoint/features.txt -mindepth 1 -maxdepth 1 -type d -iname '20*' -exec echo \"hello\" \;"

Besides eliminating the error-prone use of ls, you avoid the sub-shell and all of the issues that it brings with it.

Follow-up in response to your comment:

I’m assuming that you’re using awk -F/ '{print $NF}' to grab the name of the folder in which the file lives (that is, the last directory name before the filename). The commands basename and dirname can be used to do this for you. This should make your script a bit easier. Place the following into a script file:

#!/bin/sh
folder=$(basename $(dirname $1))
mkdir -p #{nfs_checkpoint}/${folder}
cat #{result_location}/${folder}/20* > #{nfs_checkpoint}/${folder}/features.txt

And execute it like this:

/bin/sh -c "find dir/checkpoint/features.txt -mindepth 1 -maxdepth 1 -type d -iname '20*' -exec yourscript.sh {} \;"