Home/ Questions/Q 7671335
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-31T16:01:38+00:00

I’m writing some Java code that deals with Chinese characters, and I got some

  • 0

I’m writing some Java code that deals with Chinese characters, and I got some unexpected results — strings that should be equal were not. Here is one of the offending characters, which means “six” (pinyin: liù): 六. This character can be represented with either of two code points:

F9D1 in the block: CJK Compatibility Ideographs
516D in the block: CJK Unified Ideographs

Wikipedia has a page about these character ranges, and the short section on compatibility ideographs does mention some duplicates, but the list omits this specific character.

So I’m wondering:

  1. Is there a list of duplicate unicode characters somewhere so I can transform Strings before trying to compare them?
  2. Is this normal when dealing with CJK characters, or have I done something else wrong?
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Just normalize them. U+F9D1 becomes U+516D under any of the four normalization schemes:

    $ export PERL_UNICODE=S

$ perl -le 'print "\x{F9D1}\x{516D}"' | uniquote -v
\N{CJK COMPATIBILITY IDEOGRAPH-F9D1}\N{CJK UNIFIED IDEOGRAPH-516D}

$ perl -le 'print "\x{F9D1}\x{516D}"' | nfd | uniquote -v
\N{CJK UNIFIED IDEOGRAPH-516D}\N{CJK UNIFIED IDEOGRAPH-516D}
$ perl -le 'print "\x{F9D1}\x{516D}"' | nfc | uniquote -v
\N{CJK UNIFIED IDEOGRAPH-516D}\N{CJK UNIFIED IDEOGRAPH-516D}
$ perl -le 'print "\x{F9D1}\x{516D}"' | nfkd | uniquote -v
\N{CJK UNIFIED IDEOGRAPH-516D}\N{CJK UNIFIED IDEOGRAPH-516D}
$ perl -le 'print "\x{F9D1}\x{516D}"' | nfkc | uniquote -v
\N{CJK UNIFIED IDEOGRAPH-516D}\N{CJK UNIFIED IDEOGRAPH-516D}

    Many essential Unicode tools, including those, are available here.

    • 0