Question

There is an invisible character breaking my XML parser.

c&

The XML claims to be UTF-8, but when I try to use <c:import . . . charEncoding="UTF-8">

I get this friendly message:

ERROR: javax.servlet.jsp.JspException: java.io.CharConversionException: illegal utf8 encoding at (187)

I have been able to locate the source of the problem. It is an invisible character located between ‘c’ and ‘&’.

I would like to know more about this character, but it seems IntelliJ cannot show me hidden characters . . .

I think I saw a tool online that would convert Unicode characters to their octal values, but I can not find it again. If there is a tool I need to download that would be fine.

Any suggestions?

---

OK a friend told me about od so I gave that a try:

$ echo -n "c&" | od -c
0000000    c 357 273 277 357 273 277   &                                
0000010

So it seems the problem is cause by the byte sequence 357 273 277

Do we know what that sequence is?

Answer 1

In the table below, the dots represent the breaks between octal digits, and the dashes represent the breaks between hex digits.

Octal:      3    5   7   |  2    7   3  |  2    7   7
Binary:    11.10-1.111   | 10.11-1.011  | 10.11-1.111
Hex:         E     F     |   B     B    |   B     F

This has the correct form for valid UTF-8. The first nybble shows two continuation bytes, and the next two bytes are indeed continuation bytes. The second nybble of the first byte, and the last 6 bits of each of the next two bytes form the data for the Unicode character.

Unicode Binary:  1111 1110 11.11 1111
Unicode Hex:      F     E    F    F

Therefore, the character is U+FEFF, which is the BOM (byte-order mark) or ZWNBSP (zero-width non-breaking space). It is aconventional to encode the BOM in UTF-8 (it isn’t needed); it is doubly aconventional to encode two of them in a row; and it is triply aconventional for the BOM not to be the first character in the UTF-8 code stream.

See the Unicode FAQ on BOM for more information.