Question

I’m writing a python3 program, that gets the names of files to process from command-line arguments. I’m confused regarding what is the proper way to handle different encodings.

I think I’d rather consider filenames as bytes and not strings, since that avoids the danger of using an incorrect encoding. Indeed, some of my file names use an incorrect encoding (latin1 when my system locale uses utf-8), but that doesn’t prevent tools like ls from working. I’d like my tool to be resilient to that as well.

I have two problems: the command-line arguments are given to me as strings (I use argparse), and I want to report errors to the user as strings.

I’ve successfuly adapted my code to use binaries, and my tool can handle files whose name are invalid in the current default encoding, as long as it is by recursing trough the filesystem, because I convert the arguments to binaries early, and use binaries when calling fs functions. When I receive a filename argument which is invalid, however, it is handed to me as a unicode string with strange characters like \udce8. I do not know what these are, and trying to encode it always fail, be it with utf8 or with the corresponding (wrong) encoding (latin1 here).

The other problem is for reporting errors. I expect users of my tool to parse my stdout (hence wanting to preserve filenames), but when reporting errors on stderr I’d rather encode it in utf-8, replacing invalid sequences with appropriate “invalid/question mark” characters.

So,

1) Is there a better, completely different way to do it ? (yes, fixing the filenames is planned, but i’d still like my tool to be robust)

2) How do I get the command line arguments in their original binary form (not pre-decoded for me), knowing that for invalid sequences re-encoding the decoded argument will fail, and

3) How do I tell the utf-8 codec to replace invalid, undecodable sequences with some invalid mark rather than dying on me ?

Answer 1

When I receive a filename argument
which is invalid, however, it is
handed to me as a unicode string with
strange characters like \udce8.

Those are surrogate characters. The low 8 bits is the original invalid byte.

See PEP 383: Non-decodable Bytes in System Character Interfaces.