Question

I’m currently using Java 6 (I don’t have the option of moving to Java 7) and I’m trying to use the java.util.regex package to do pattern matching of strings that contain Unicode characters.

I know that java.lang.String supports supplemental characters (i.e. characters with codepoints > 0xFFFF) (since Java 5), but I don’t see a simple way to do do pattern matching with these characters. java.util.regex.Pattern still only allows hexadecimals to be represented using 4 digits (e.g. \uFFFF)

Does anyone know if I’m missing an API here?

Answer 1

I’ve never done pattern matching with supplemental characters, but I think it’s as simple as encoding them (in patterns and strings) as two 16 bits numbers (a UTF-16 surrogate pair) \unnnn\ummmm . java.util.regex should be is clever enough to interpret those two numbers (Java chars) as a single character in patterns and strings (though Java will still see them as two chars, as elements of the string).

Two links:

Java Unicode encoding

http://java.sun.com/developer/technicalArticles/Intl/Supplementary/

From the last link (refering to Java 5) :

The java.util.regex package has been
updated so that both pattern strings
and target strings can contain
supplementary characters, which will
be handled as complete units.

Note also that, if you are using UTF8 as your encoding (for your source files), you can also write them directly (see section “Representing Supplementary Characters in Source Files” in the last link).

For example:

    String pat1 = ".*\uD840\uDC00{2}.*";
    String s1  = "HI \uD840\uDC00\uD840\uDC00 BYE";
    System.out.println(s1.matches(pat1) + " len=" + s1.length());

    String pat2 = ".*\u0040\u0041{2}.*";
    String s2 = "HI \u0040\u0041\u0040\u0041 BYE";
    System.out.println(s2.matches(pat2) + " len=" + s2.length());

This, compiled with Java 6, prints

true len=11
false len=11

which agrees with the above. In the first case, we have a single code point, represented as a pair of surrogate java chars (two 16 bits chars, one suplemental Unicode character), and the {2} quantifier applies to the pair(=codepoint). In the second, we have two distinct BMP characters, the quantifier applies to the last one – hence, no match.

Notice, however, that the string length is the same (because Java measures the string length counting Java characters, not Unicode code points).