Question

Could you explain in detail what the difference is between byte string and Unicode string in Python. I have read this:

Byte code is simply the converted source code into arrays of bytes

Does it mean that Python has its own coding/encoding format? Or does it use the operation system settings?
I don’t understand. Could you please explain?
Thank you!

Answer 1

No, Python does not use its own encoding – it will use any encoding that it has access to and that you specify.

A character in a str represents one Unicode character. However, to represent more than 256 characters, individual Unicode encodings use more than one byte per character to represent many characters.

bytes objects give you access to the underlying bytes. str objects have the encode method that takes a string representing an encoding and returns the bytes object that represents the string in that encoding. bytes objects have the decode method that takes a string representing an encoding and returns the str that results from interpreting the byte as a string encoded in the the given encoding.

For example:

>>> a = "αά".encode('utf-8')
>>> a
b'\xce\xb1\xce\xac'
>>> a.decode('utf-8')
'αά'

We can see that UTF-8 is using four bytes, \xce, \xb1, \xce, and \xac, to represent two characters.

Related reading:

• Python Unicode Howto (from the official documentation)

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

• Pragmatic Unicode by Ned Batchelder