Question

I recently realized that I don’t fully understand Java’s string encoding process.

Consider the following code:

public class Main
{
    public static void main(String[] args)
    {
        System.out.println(java.nio.charset.Charset.defaultCharset().name());
        System.out.println("ack char: ^"); /* where ^ = 0x06, the ack char */
    }
}

Since the control characters are interpreted differently between windows-1252 and ISO-8859-1, I chose the ack char for testing.

I now compile it with different file encodings, UTF-8, windows-1252, and ISO-8859-1. The both compile to the exact same thing, byte-per-byte as verified by md5sum.

I then run the program:

$ java Main | hexdump -C
00000000  55 54 46 2d 38 0a 61 63  6b 20 63 68 61 72 3a 20  |UTF-8.ack char: |
00000010  06 0a                                             |..|
00000012

$ java -Dfile.encoding=iso-8859-1 Main | hexdump -C
00000000  49 53 4f 2d 38 38 35 39  2d 31 0a 61 63 6b 20 63  |ISO-8859-1.ack c|
00000010  68 61 72 3a 20 06 0a                              |har: ..|
00000017

$ java -Dfile.encoding=windows-1252 Main | hexdump -C
00000000  77 69 6e 64 6f 77 73 2d  31 32 35 32 0a 61 63 6b  |windows-1252.ack|
00000010  20 63 68 61 72 3a 20 06  0a                       | char: ..|
00000019

It correctly outputs the 0x06 no matter which encoding is being used.

Ok, it still outputs the same 0x06, which would be interpreted as the printable [ACK] char by windows-1252 codepages.

That leads me to a few questions:

1. Is the codepage / charset of the Java file being compiled expected to be identical to the default charset of the system under which it’s being compiled? Are the two always synonymous?
2. The compiled representation doesn’t seem dependent on the compile-time charset, is this indeed the case?
3. Does this imply that strings within Java files may be interpreted differently at runtime if they don’t use standard characters for the current charset/locale?
4. What else should I really know about string and character encoding in Java?

Answer 1

1. Source files can be in any encoding
2. You need to tell the compiler the encoding of source files (e.g. javac -encoding...); otherwise, platform encoding is assumed
3. In class file binaries, string literals are stored as (modified) UTF-8, but unless you work with bytecode, this doesn’t matter (see JVM spec)
4. Strings in Java are UTF-16, always (see Java language spec)
5. The System.out PrintStream will transform your strings from UTF-16 to bytes in the system encoding prior to writing them to stdout

Notes:

• Blog post I wrote on Java encoding
• Don’t use -Dfile.encoding