Question

In my Python script running at the command line on Ubuntu, it’s selecting UTF8-encoded content from a MySQL database.

Then, I want to print the string to the console.

The displayed strings have an encoding problem, as they don’t show the accented characters correctly. How do I fix this?

Preferably, the script would make the decision, rather than setting a system environment, to ensure easy running on other systems.

Answer 1

It is very strongly recommended that you not use “?” as a replacement char. Just set your output encoding to UTF-8 and be done with it.

for s in ("stdin","stdout","stderr"): 
   setattr(sys, s, io.TextIOWrapper(getattr(sys, s).detach(), encoding="utf8"))

Alternately, set your PYTHONIOENCODING envariable to utf8 so that python stops guessing about the output encoding.

Either approach is infinitely much better than manually encoding, which is stupid.

If you refuse to upgrade to Python3, I also recommend

from __future__ import unicode_literals

to banish all that stupid u'...' stuff.

Lately I’ve starting all my Python progams like this:

#!/usr/bin/env python3.2
# -*- coding: UTF-8 -*-

from __future__ import print_function
from __future__ import unicode_literals

import re
import sys
import os

if not (("PYTHONIOENCODING" in os.environ)
            and
        re.search("^utf-?8$", os.environ["PYTHONIOENCODING"], re.I)):
    sys.stderr.write(sys.argv[0] + ": Please set your PYTHONIOENCODING envariable to utf8\n")
    sys.exit(1)

import unicodedata
if unicodedata.unidata_version < "6.0.0":
    print("WARNING: Your old UCD is out of date, expected at least 6.0.0 but got", 
           unicodedata.unidata_version)

wide_enough = (sys.maxunicode >= 0x10FFFF)
if not wide_enough:
    print("WARNING: Narrow build detected, your Python lacks full Unicode support!!")