Question

Problem

I’m writing a set of scripts to help with automated batch job execution on a cluster.
The specific thing I have is a $OUTPUT_DIR, and an arbitrary $COMMAND.
I would like to execute the $COMMAND such that its output ends up in $OUTPUT_DIR.
For example, if COMMAND='cp ./foo ./bar; mv ./bar ./baz', I would like to run it such that the end result is equivalent to cp ./foo ./$OUTPUT_DIR/baz.
Ideally, the solution would look something like eval PWD="./$OUTPUT_DIR" $COMMAND, but that doesn’t work.

Known solutions

[And their problems]

• Editing $COMMAND: In most cases the command will be a script, or a compiled C or FORTRAN executable. Changing the internals of these isn’t an option.
• unionfs, aufs, etc.: While this is basically perfect, users running this won’t have root, and causing thousands+ of arbitrary mounts seems like a questionable choice.
• copying/ hard/soft links: This might be the solution I will have to use: some variety of actually duplicating the entire content of ./ into ./$OUTPUT_DIR
• cd $OUTPUT_DIR; ../$COMMAND : Fails if $COMMAND ever reads files
• pipes : only works if $COMMAND doesn’t directly work with files; which it usually does

Is there another solution that I’m missing, or is this request actually impossible?

[EDIT:]Chosen Solution

I’m going to go with something where each object in the directory is symbolic-linked into the output directory, and the command is then run from there.
This has the downside of creating a lot of symbolic links, but it shouldn’t be too bad.

Answer 1

You can’t solve this without making some assumptions about the interface of $COMMAND. There is no single definition of what “output ends up in $OUTPUT_DIR” means. For one program this may be some files, but another program might just print something to stdout and yet another might try sending some data over the internet using some protocol or display something in a GUI and there isn’t an obvious way of mapping all of these to “output goes to $OUTPUT_DIR“.

So, you need to invent some assumptions and require any $COMMAND implementation to follow them. Then, it may get as simple as requesting that the command accept a parameter such as --target=<DIR>. If your command was some simple command, you would have to create a wrapper script around it to translate that parameter into what the app accepts. cp, mv and a few more utils already accept the parameter --target, so that may be a good starting point.