Question

I’ve noticed the length method of std::string returns the length in bytes and the same method in std::u16string returns the number of 2-byte sequences.

I’ve also noticed that when a character or code point is outside of the BMP, length returns 4 rather than 2.

Furthermore, the Unicode escape sequence is limited to \unnnn, so any code point above U+FFFF cannot be inserted by an escape sequence.

In other words, there doesn’t appear to be support for surrogate pairs or code points outside of the BMP.

Given this, is the accepted or recommended practice to use a non-standard string manipulation library that understands UTF-8, UTF-16, surrogate pairs, and so on?

Does my complier have a bug or am I using the standard string manipulation methods incorrectly?

Example:

/*
* Example with the Unicode code points U+0041, U+4061, U+10196 and U+10197
*/

#include <iostream>
#include <string>

int main(int argc, char* argv[])
{
    std::string example1 = u8"A䁡";
    std::u16string example2 = u"A䁡";

    std::cout << "Escape Example: " << "\u0041\u4061\u10196\u10197" << "\n";
    std::cout << "Example: " << example1 << "\n";
    std::cout << "std::string Example length: " << example1.length() << "\n";
    std::cout << "std::u16string Example length: " << example2.length() << "\n";

    return 0;
}

Here is the result I get when compiled with GCC 4.7:

Escape Example: A䁡မ6မ7
Example: A䁡
std::string Example length: 12
std::u16string Example length: 6

Answer 1

At the risk of judging prematurely, it seems to me that the language used in the standards in slightly ambiguous (although the final conclusion is clear, see at the end):

In the description of char16_t literals (i.e. the u"..." ones like in your example), the size of a literal is defined as:

The size of a char16_t string literal is the total number of escape sequences, universal-character-names, and other characters, plus one for each character requiring a surrogate pair, plus one for the terminating u’\0’.

And the footnote further clarifies:

[ Note: The size of a char16_t string literal is the number of code units, not the number of characters. —end note ]

This implies a definition of character and code unit. A surrogate pair is one
character, but two code units.

However, the description of the length() method of std::basic_string (of which std::u16string is derived) claims:

Returns the number of characters in the string, i.e. std::distance(begin(), end()). It is the same as size().

As it appears, the description of length() uses the word character to mean what the definition of char16_t calls a code unit.

However, the conclusion of all of this is: The length is defined as code units, hence your compiler complies with the standard, and there will be continued demand for special libraries to provide proper counting of characters.

I used the references below:

• For the definition of the size of char16_t literals: Here
• For the description of std::basic_string::length(): Here