Question

When running a Python program that reads from stdin, I get the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 320: ordinal not in range(128)

How can I fix it?

Note: The error occurs internal to antlr and the line looks like that:

        self.strdata = unicode(data)

Since I don’t want to modify the source code,
I’d like to pass in something that is acceptable.

The input code looks like that:

#!/usr/bin/python
import sys
import codecs
import antlr3
import antlr3.tree
from LatexLexer import LatexLexer
from LatexParser import LatexParser


char_stream = antlr3.ANTLRInputStream(codecs.getreader("utf8")(sys.stdin))
lexer = LatexLexer(char_stream)
tokens = antlr3.CommonTokenStream(lexer)
parser = LatexParser(tokens)
r = parser.document()

Answer 1

The problem is, that when reading from stdin, python decodes
it using the system default encoding:

>>> import sys
>>> sys.getdefaultencoding()
'ascii'

The input is very likely UTF-8 or Windows-CP-1252, so the program
chokes on non-ASCII-characters.

To convert sys.stdin to a stream with the proper decoder, I used:

import codecs
char_stream = codecs.getreader("utf-8")(sys.stdin)

That fixed the problem.

BTW, this is the method ANTLRs FileStream uses to open a file
with given filename (instead of a given stream):

    fp = codecs.open(fileName, 'rb', encoding)
    try:
        data = fp.read()
    finally:
        fp.close()

BTW #2: For strings I found

a_string.encode(encoding) 

useful.