Question

I’m running ./sample.py --url http://blah.com without error, though if I run ./sample.py --url http://blah.com | wc -l or similar I receive an error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u200f' in position 0: ordinal not in range(128)

How do I make a python script compatible with my terminal commands? I keep seeing reference to sys.stdin.isatty though its use case appears to be opposite.

Answer 1

When Python detects that it is printing to a terminal, sys.stdout.encoding is set to the encoding of the terminal. When you print a unicode, the unicode is encoded to a str using the sys.stdout.encoding.

When Python does not detect that it is printing to a terminal, sys.stdout.encoding is set to None. When you print a unicode, the ascii codec is used (at least in Python2). This will result in a UnicodeError if the unicode contains code points outside of 0-127.

One way to fix this is to explicitly encode your unicode before printing. That perhaps is the proper way, but it can be laborious if you have a lot of print statements scattered around.

Another way to fix this is to set the PYTHONIOENCODING environment variable to an appropriate encoding. For example,

PYTHONIOENCODING=utf-8

Then this encoding will be used instead of ascii when printing output to a file.

See the PrintFails wiki page for more information.