Question

When I’m writing shell scripts, I often find myself spending most of my time (especially when debugging) dealing with argument processing. Many scripts I write or maintain are easily more than 80% input parsing and sanitization. I compare that to my Python scripts, where argparse handles most of the grunt work for me, and lets me easily construct complex option structures and sanitization / string parsing behavior.

I’d love, therefore, to be able to have Python do this heavy lifting, and then get these simplified and sanitized values in my shell script, without needing to worry any further about the arguments the user specified.

To give a specific example, many of the shell scripts where I work have been defined to accept their arguments in a specific order. You can call start_server.sh --server myserver --port 80 but start_server.sh --port 80 --server myserver fails with You must specify a server to start. – it makes the parsing code a lot simpler, but it’s hardly intuitive.

So a first pass solution could be something as simple as having Python take in the arguments, sort them (keeping their parameters next to them) and returning the sorted arguments. So the shell script still does some parsing and sanitization, but the user can input much more arbitrary content than the shell script natively accepts, something like:

# script.sh -o -aR --dir /tmp/test --verbose

#!/bin/bash

args=$(order.py "$@")
# args is set to "-a --dir /tmp/test -o -R --verbose"

# simpler processing now that we can guarantee the order of parameters

There’s some obvious limitations here, notably that parse.py can’t distinguish between a final option with an argument and the start of indexed arguments, but that doesn’t seem that terrible.

So here’s my question: 1) Is there any existing (Python preferably) utility to enable CLI parsing by something more powerful than bash, which can then be accessed by the rest of my bash script after sanitization, or 2) Has anyone done this before? Are there issues or pitfalls or better solutions I’m not aware of? Care to share your implementation?

---

One (very half-baked) idea:

#!/bin/bash

# Some sort of simple syntax to describe to Python what arguments to accept
opts='
"a", "append", boolean, help="Append to existing file"
"dir", str, help="Directory to run from"
"o", "overwrite", boolean, help="Overwrite duplicates"
"R", "recurse", boolean, help="Recurse into subdirectories"
"v", "verbose", boolean, help="Print additional information"
'

# Takes in CLI arguments and outputs a sanitized structure (JSON?) or fails
p=$(parse.py "Runs complex_function with nice argument parsing" "$opts" "$@")
if [ $? -ne 0 ]; exit 1; fi # while parse outputs usage to stderr

# Takes the sanitized structure and an argument to get
append=$(arg.py "$p" append)
overwrite=$(arg.py "$p" overwrite)
recurse=$(arg.py "$p" recurse)
verbose=$(arg.py "$p" verbose)

cd $(python arg.py "$p" dir)

complex_function $append $overwrite $recurse $verbose

Two lines of code, along with concise descriptions of the arguments to expect, and we’re on to the actual script behavior. Maybe I’m crazy, but that seems way nicer than what I feel like I have to do now.

---

I’ve seen Parsing shell script arguments and things like this wiki page on easy CLI argument parsing, but many of these patterns feel clunky and error prone, and I dislike having to re-implement them every time I write a shell script, especially when Python, Java, etc. have such nice argument processing libraries.

Answer 1

Edit: I haven’t used it (yet), but if I were posting this answer today I would probably recommend https://github.com/docopt/docopts instead of a custom approach like the one described below.

---

I’ve put together a short Python script that does most of what I want. I’m not convinced it’s production quality yet (notably error handling is lacking), but it’s better than nothing. I’d welcome any feedback.

It takes advantage of the set builtin to re-assign the positional arguments, allowing the remainder of the script to still handle them as desired.

bashparse.py

#!/usr/bin/env python

import optparse, sys
from pipes import quote

'''
Uses Python's optparse library to simplify command argument parsing.

Takes in a set of optparse arguments, separated by newlines, followed by command line arguments, as argv[2] and argv[3:]
and outputs a series of bash commands to populate associated variables.
'''

class _ThrowParser(optparse.OptionParser):
    def error(self, msg):
        """Overrides optparse's default error handling
        and instead raises an exception which will be caught upstream
        """
        raise optparse.OptParseError(msg)

def gen_parser(usage, opts_ls):
    '''Takes a list of strings which can be used as the parameters to optparse's add_option function.
    Returns a parser object able to parse those options
    '''
    parser = _ThrowParser(usage=usage)
    for opts in opts_ls:
        if opts:
            # yes, I know it's evil, but it's easy
            eval('parser.add_option(%s)' % opts)
    return parser

def print_bash(opts, args):
    '''Takes the result of optparse and outputs commands to update a shell'''
    for opt, val in opts.items():
        if val:
            print('%s=%s' % (opt, quote(val)))
    print("set -- %s" % " ".join(quote(a) for a in args))

if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.stderr.write("Needs at least a usage string and a set of options to parse")
        sys.exit(2)
    parser = gen_parser(sys.argv[1], sys.argv[2].split('\n'))

    (opts, args) = parser.parse_args(sys.argv[3:])
    print_bash(opts.__dict__, args)

Example usage:

#!/bin/bash

usage="[-f FILENAME] [-t|--truncate] [ARGS...]"
opts='
"-f"
"-t", "--truncate",action="store_true"
'

echo "$(./bashparse.py "$usage" "$opts" "$@")"
eval "$(./bashparse.py "$usage" "$opts" "$@")"

echo
echo OUTPUT

echo $f
echo $@
echo $0 $2

Which, if run as: ./run.sh one -f 'a_filename.txt' "two' still two" three outputs the following (notice that the internal positional variables are still correct):

f=a_filename.txt
set -- one 'two'"'"' still two' three

OUTPUT
a_filename.txt
one two' still two three
./run.sh two' still two

Disregarding the debugging output, you’re looking at approximately four lines to construct a powerful argument parser. Thoughts?