Question

While writing an encryption method in JavaScript, I came to wondering what character encoding my strings were using, and why.

What determines character encoding in JavaScript? Is it a standard? By the browser? Determined by the header of the HTTP request? In the <META> tag of HTML that encompasses it? The server that feeds the page?

By my empirical testing (changing different settings, then using charCodeAt on a sufficiently strange character and seeing which encoding the value matches up with) it appears to always be UTF-8 or UTF-16, but I’m not sure why.

_{After some frantic googling, I couldn’t seem to find a conclusive answer to this simple question.}

Answer 1

Section 8.4 of E262:

The String type is the set of all finite ordered sequences of zero or more 16-bit unsigned integer values (“elements”). The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a code unit value (see Clause 6). Each element is regarded as occupying a position within the sequence. These positions are indexed with nonnegative integers. The first element (if any) is at position 0, the next element (if any) at position 1, and so on. The length of a String is the number of elements (i.e., 16-bit values) within it. The empty String has length zero and therefore contains no elements.

When a String contains actual textual data, each element is considered to be a single UTF-16 code unit. Whether or not this is the actual storage format of a String, the characters within a String are numbered by their initial code unit element position as though they were represented using UTF-16. All operations on Strings (except as otherwise stated) treat them as sequences of undifferentiated 16-bit unsigned integers; they do not ensure the resulting String is in normalised form, nor do they ensure language-sensitive results.

That wording is kind of weaselly; it seems to mean that everything that counts treats strings as if each character is a UTF-16 character, but at the same time nothing ensures that it’ll all be valid.

To be clear, the intention is that strings consist of UTF-16 code points. In ES2015, the definition of "string value" includes this note:

A String value is a member of the String type. Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text. However, ECMAScript does not place any restrictions or requirements on the values except that they must be 16-bit unsigned integers.

So a string is still a string even when it contains values that don’t work as correct Unicode characters.