Question

I’m a bit puzzled by the way paths are handled in Python. Using common constructs line “~” or “.” or “..”, I often run into cases where a path is not recognized as valid or existing, especially if I pass the path on as an argument to a shell command; but all of my problems go away if I always do something like:

some_path = os.path.abspath(os.path.expanduser(some_path))

Is this a common — or perhaps even required — idiom, or am I just reinventing the wheel? Should I really expect that wherever I have some_path, I should have the above code before passing it to any (or at least most) functions that do anything with it?

Answer 1

Yes, most things you can call will expect path that has been run through that idiom. When you use paths like that in the shell (eg, when you do something like cat ~raxacoricofallapatorius/foo.txt), it is the shell itself – rather than cat or any other program you might run – that does the path normalisation.

You can verify this trivially – eg,

lvc@tiamat:~/Projects$ echo ~
/home/lvc

So this does mean that if you expect to get a path with those kinds of variables as input, you will need to do the preprocessing yourself. The alternative is to run the commands through a shell, and be ready to deal with all the problems that brings.

However, at least on unix-like systems (Windows may or may not behave the same way), you don’t need to do this for . and .. – these are understood by the system calls, and the shell does not transform them – so, eg:

lvc@tiamat:~/Projects$ file ..
..: directory
lvc@tiamat:~/Projects$ file ~
/home/lvc: directory

Notice that file sees .. unchanged, but sees the expanded form of ~.

This means that if all you want is paths that will work directly in external programs, passing it through expanduser and possibly expandvars is sufficient. You will want to call abspath primarily if the program you are calling out to will run in a different working directory than yours.