Question

As many hopefully can relate, this encoding problem is driving me mental. I would really appreciate some light on this!

End goal is to be able to run the same script.py from both terminal and cron, and from cron with > stdout.txt. And needless to say, I’m having serious encoding troubles.

My script.py runs fine from terminal thus: python script.py

It throws an error, however, when run from terminal thus: python script.py > stdout.txt

It throws same error when run in cron, either way.

I have a python script, entered in crontab -e as root.

This is my script.py header:

#!/usr/bin/python
# -*- coding: utf-8 -*-

This is my cron entry:

* * * * * python /home/ubuntu/parrot/script.py > /home/ubuntu/parrot/stdout.txt

This is my stdout.txt (the relevant part):

Unexpected error!  (<type 'exceptions.UnicodeDecodeError'>, UnicodeDecodeError('ascii', 'blabla some weird text n\xc3\xa5r end', 54, 55, 'ordinal not in range(128)')) 

This is my env from terminal (the relevant part):

LANG=en_US.UTF-8

This is my env from cron (the relevant part):

LANG=en_US.UTF-8

This is the (first) line in script.py throwing the error:

print 'Posting @%s: %s' % (statusObj.user.screen_name.encode('ascii', 'replace'), statusObj.text.encode('utf-8', 'replace'))

Edit:
sys.getdefaultencoding() returns ascii

Any help is greatly appreciated!

Answer 1

If you have control over the statusObj you should check the relevant code where data is being parsed into the object and try to get the input as clean as possible.

You want to make sure your string is decoded to unicode before you try to encode it.

If not you can try:

# try to get the string into unicode
screen_name = unicode(statusObj.user.screen_name) 
post = unicode(statusObj.text) # probably an error here?
output_str = u"Posting @{name}: {post}".format(name=screen_name, post=post)
print output_str.encode("utf8", "replace") # encode the unicode string on