Recently I hit a bug due to data quality with browser support, and I am looking for a safe rule for applying string escape without double size unless required.
A UTF-8 byte sequence “E2-80-A8” (U+2028, LINE SEPARATOR), a perfectly valid character in a Unicode database. However, that sequence represents a line-separator (Yes, other then “0A”).
And badly, many browser (including Chrome, Firefox, and Safari; I didn’t test others), failed to process a JSONP callback which has a string that contains that Unicode character. The JSONP was included by a non-Unicode HTML which I did not have any control.
The browsers simply reported INVALID CODE/syntax error on such JavaScript which looks valid from debug tools and all text editors. What I guess is that it may try to convert “E2-80-A8” to BIG-5 and broke JS syntax.
The above is only an example of how Unicode can break your system unexpected. As far as I know, some hacker can use RTL and other control characters for their good. And there are many “quotes”, “spaces”, “symbols” and “controls” in Unicode specification.
QUESTION:
Is there a list of Unicode characters for every programmer to know about hidden features (and bugs) which we might not want them effective in our application. (e.g. Windows disable RTL in filename).
EDIT:
I am not asking for JSON nor JavaScript. I am asking for general best practice of Unicode handing in all programs.
There’s a database of character properties and a report describing it, the UNICODE CHARACTER DATABASE, that gives a good idea of how browsers “should” treat a code point. I love that word, “should”. Safest is going to be a whitelist, you could probably go with L|M|N|S, Letter or Mark or Number or Symbol.
Have a look at the ICU project for a library