Question

In my application I have to constantly convert string between std::string and std::wstring due different APIs (boost, win32, ffmpeg etc..). Especially with ffmpeg the strings end up utf8->utf16->utf8->utf16, just to open a file.

Since UTF8 is backwards compatible with ASCII I thought that I consistently store all my strings UTF-8 std::string and only convert to std::wstring when I have to call certain unusual functions.

This worked kind of well, I implemented to_lower, to_upper, iequals for utf8. However then I met several dead-ends std::regex, and regular string comparisons. To make this usable I would need to implement a custom ustring class based on std::string with re-implementation of all corresponding algorithms (including regex).

Basically my conclusion is that utf8 is not very good for general usage. And the current std::string/std::wstring is mess.

However, my question is why the default std::string and "" are not simply changed to use UTF8? Especially as UTF8 is backward compatible? Is there possibly some compiler flag which can do this? Of course the stl implemention would need to be automatically adapted.

I’ve looked at ICU, but it is not very compatible with apis assuming basic_string, e.g. no begin/end/c_str etc…

Answer 1

The main issue is the conflation of in-memory representation and encoding.

None of the Unicode encoding is really amenable to text processing. Users will in general care about graphemes (what’s on the screen) while the encoding is defined in terms of code points… and some graphemes are composed of several code points.

As such, when one asks: what is the 5th character of "Hélène" (French first name) the question is quite confusing:

• In terms of graphemes, the answer is n.
• In terms of code points… it depends on the representation of é and è (they can be represented either as a single code point or as a pair using diacritics…)

Depending on the source of the question (a end-user in front of her screen or an encoding routine) the response is completely different.

Therefore, I think that the real question is Why are we speaking about encodings here?

Today it does not make sense, and we would need two “views”: Graphemes and Code Points.

Unfortunately the std::string and std::wstring interfaces were inherited from a time where people thought that ASCII was sufficient, and the progress made didn’t really solve the issue.

I don’t even understand why the in-memory representation should be specified, it is an implementation detail. All a user should want is:

• to be able to read/write in UTF-* and ASCII
• to be able to work on graphemes
• to be able to edit a grapheme (to manage the diacritics)

… who cares how it is represented? I thought that good software was built on encapsulation?

Well, C cares, and we want interoperability… so I guess it will be fixed when C is.