Question

We write a C++ application and need to know this:

Is UTF8 text encoding an injective mapping from bytes to characters, meaning that every single character (letter…) is encoded in only one way? So, e.g. letter ‘Ž’ cannot be encoded as, say, both 3231 and 32119.

Answer 1

That depends very much on what you consider a “letter”.

UTF8 is basically a tiny piece of what is Unicode.

Basically there are at least three levels: Bytes, Code points and Grapheme clusters.
A Code point can be encoded in one or more bytes, according to a certain encoding, like UTF8, UTF16 or UTF32. This encoding is unique (because all alternative ways are declared invalid). However a code point is not always a glyph because there are so-called combining characters. Such combining characters follow the base character and, as their name says, are combined with the base character. For example, there’s the combining character U+0308 COMBINING DIAERESIS which puts a diaeresis (¨) above the preceding letter. So if it follows e.g. an a (U+0061 LATIN SMALL LETTER A), the result is an ä. However there’s also a single code point for the letter ä (U+00E4 LATIN SMALL LETTER A WITH DIAERESIS), so this means that the code sequences U+0061 U+0308 and U+00E4 describe the same letter.

So, each code point has a single valid UTF 8 encoding (e.g. U+0061 is “\141”, U+0308 is “\314\210” and U+00e4 is “\303\244”, but the letter ä is encoded by both the code point sequence U+0061 U+0308, i.e. in UTF8 the byte sequence “\141\314\210” and the single code point U+00E4, i.e. the byte sequence “\303\244”.

What’s worse is that since the Unicode makers decided that the combining letters follow the base letter instead of preceding it, you cannot know whether your glyph is complete until you’ve seen the next code point (if it is not a combining code point, your letter is finished).