I have an html box with which users may enter text. I would like to ensure all text entered in the box is either encoded in UTF-8 or converted to UTF-8 when a user finishes typing. Furthermore, I don’t quite understand how various UTF encoding are chosen when being entered into a text box.
Generally I’m curious about the following:
- How does a browser determine which encodings to use when a user is typing into a text box?
- How can javascript determine the encoding of a string value in an html text box?
- Can I force the browser to only use UTF-8 encoding?
- How can I encode arbitrary encodings to UTF-8 I assume there is a JavaScript library for this?
** Edit **
Removed some questions unnecessary to my goals.
This tutorial helped me understand JavaScript character codes better, but is buggy and does not actually translate character codes to utf-8 in all cases.
http://www.webtoolkit.info/javascript-base64.html
It uses the encoding the page was decoded as by default. According to the spec, you should be able to override this with the
accept-charsetattribute of the<form>element, but IE is buggy, so you shouldn’t rely on this (I’ve seen several different sources describe several different bugs, and I don’t have all the relevant versions of IE in front of me to test, so I’ll leave it at that).All strings in JavaScript are encoded in UTF-16. The browser will map everything into UTF-16 for JavaScript, and from UTF-16 into whatever the page is encoded in.
UTF-16 is an encoding that grew out of UCS-2. Originally, it was thought that 65,536 code points would be enough for all of Unicode, and so a 16 bit character encoding would be sufficient. It turned out that the is not the case, and so the character set was expanded to 1,114,112 code points. In order to maintain backwards compatibility, a few unused ranges of the 16 bit character set were set aside for surrogate pairs, in which two 16 bit code units were used to encode a single character. Read up on UTF-16 and UCS-2 on Wikipedia for details.
The upshot is that when you have a string
strin JavaScript,str.lengthdoes not give you the number of characters, it gives you the number of code units, where two code units may be used to encode a single character, if that character is not within the Basic Multilingual Plane. For instance,"abc".lengthgives you 3, but"".lengthgives you 6; and"".substring(0,1)gives what looks like an empty string, since a half of a surrogate pair cannot be displayed, but the string still contains that invalid character (I will not guarantee this works cross browser; I believe it is acceptable to drop broken characters). To get a valid character, you must use"".substring(0,2).The best way to do this is to deliver your page in UTF-8. Ensure that your web server is sending the appropriate
Content-type: text/html; charset=UTF-8headers. You may also want to embed a<meta charset="UTF-8">element in your<head>element, for cases in which theContent-Typedoes not get set properly (such as if your page is loaded off of the local disk).There isn’t much need in JavaScript to encode text in particular encodings. If you are simply writing to the DOM, or reading or filling in form controls, you should just use JavaScript strings which are treated as sequences of UTF-16 code units.
XMLHTTPRequest, when used tosend(data)via POST, will use UTF-8 (if you pass it a document with a different encoding declared in the<?xml ...>declaration, it may or may not convert that to UTF-8, so for compatibility you generally shouldn’t use anything other than UTF-8).