Question

Probably I completely don’t understand it, so can you take a look at code examples and tell my what should I do, to be sure it will work?

I tried it in Eclipse with Pydev. I use python 2.6.6 (becuase of some library that not support python 2.7).

First, without using codecs module

# -*- coding: utf-8 -*-

file1 = open("samoloty1.txt", "w")
file2 = open("samoloty2.txt", "w")
file3 = open("samoloty3.txt", "w")
file4 = open("samoloty4.txt", "w")
file5 = open("samoloty5.txt", "w")
file6 = open("samoloty6.txt", "w")

# I know that this is weird, but it shows that whatever i do, it not ruin anything...
print u"ą✈✈"
file1.write(u"ą✈✈")
print "ą✈✈"
file2.write("ą✈✈")

print "ą✈✈".decode("utf-8")
file3.write("ą✈✈".decode("utf-8"))
print "ą✈✈".encode("utf-8")
file4.write("ą✈✈".encode("utf-8"))

print u"ą✈✈".decode("utf-8")
file5.write(u"ą✈✈".decode("utf-8"))
print u"ą✈✈".encode("utf-8")
file6.write(u"ą✈✈".encode("utf-8"))

file1.close()
file2.close()
file3.close()
file4.close()
file5.close()
file6.close()

file1 = open("samoloty1.txt", "r")
file2 = open("samoloty2.txt", "r")
file3 = open("samoloty3.txt", "r")
file4 = open("samoloty4.txt", "r")
file5 = open("samoloty5.txt", "r")
file6 = open("samoloty6.txt", "r")

print file1.read()
print file2.read()
print file3.read()
print file4.read()
print file5.read()
print file6.read()

Every each of those prints works correctly and I don’t get any funny characters.

Also i tried this: i delete all files made in the previous test and change only those lines:

file1 = open("samoloty1.txt", "w")

to those:

file1 = codecs.open("samoloty1.txt", "w", encoding='utf-8')

and again everything works…

Can anyone make some examples what works, and what not?

Should this be separate question?
I am downloading web pages, through this:

content = urllib.urlopen(some_url).read()
ucontent = unicode(content, encoding) # i get encoding from headers

Is this correct and enough? What should I do next with it to store it in utf-8 file? (I ask it because whatever I did before, it just works…)

** UPDATE **

Probably everything works ok because PyDev (or just Eclipse) has terminal encoded in UTF-8. So for tests i used cmd from Windows 7 and i get some errors. Now everything was crashing as expected. 😀 Here i am showing what i changed to get it working again (and all of those changes are reasonable for me and they agree with what i learn in answers and in docs in Python documentations).

print u"ą✈✈".encode("utf-8") # added encode
file1.write(u"ą✈✈".encode("utf-8")) # added encode
print "ą✈✈"
file2.write("ą✈✈")

print "ą✈✈" # removed .decode("utf-8")
file3.write("ą✈✈") # removed .decode("utf-8"))
print "ą✈✈" # removed .encode("utf-8")
file4.write("ą✈✈") # removed .encode("utf-8"))

print u"ą✈✈".encode("utf-8") # changed from .decode("utf-8")
file5.write(u"ą✈✈".encode("utf-8")) # changed from .decode("utf-8")
print u"ą✈✈".encode("utf-8")
file6.write(u"ą✈✈".encode("utf-8"))

And like someone said, when i use codecs, i not need to use encode() everytime before writing to file. 🙂
Question is, which answer should be marked as correct?

Answer 1

You are just lucky that the encoding of your console is utf-8 by default.

If you pass a unicode object to the write method method of a file object (sys.stdout) the object is implicitly decoded with its encoding attribute.

Thouse who work in Windows are not so lucky: How to workaround Python "WindowsError messages are not properly encoded" problem?