Question

I’ve got a python script that outputs unicode to the console, and I’d like to redirect it to a file. Apparently, the redirect process in python involves converting the output to a string, so I get errors about inability to decode unicode characters.

So then, is there any way to perform a redirect into a file encoded in UTF-8?

Answer 1

When printing to the console, Python looks at sys.stdout.encoding to determine the encoding to use to encode unicode objects before printing.

When redirecting output to a file, sys.stdout.encoding is None, so Python2 defaults to the ascii encoding. (In contrast, Python3 defaults to utf-8.) This often leads to an exception when printing unicode.

You can avoid the error by explicitly encoding the unicode yourself before printing:

print (unicode_obj.encode('utf-8'))

or you could redefine sys.stdout so all output is encoded in utf-8:

import sys
import codecs
sys.stdout=codecs.getwriter('utf-8')(sys.stdout)
print(unicode_obj)