Question

I’m struggling to understand the behaviour of regular expressions in Java and have encountered something that seems very strange. In the code below, the test fails all of a sudden for reasons I don’t understand at the test with the message label “6 letters matched, negative” (the subsequent two tests fail as well). Have I been staring at this for too long or is there indeed something strange happening? I don’t this has to do with the variable-length negative lookahead assertion (?!X), but I’d be happy to hear any theories, or even a confirmation that others are experiencing the same issue and that it’s not specific to my JVM. Sorry that the regular expression is so contrived, but you don’t want to see the real thing 🙂

// $ java -version
// java version "1.7.0_10"
// Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
// Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)

// test of word without agreement
String test = "plusieurs personne sont";

// match the pattern with curly braces
assertTrue("no letters matched", Pattern.compile("plusieurs personne\\b").matcher(test).find());
assertTrue("1 letters matched", Pattern.compile("plusieurs personn\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("2 letters matched", Pattern.compile("plusieurs person\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("3 letters matched", Pattern.compile("plusieurs perso\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("4 letters matched", Pattern.compile("plusieurs pers\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("5 letters matched", Pattern.compile("plusieurs per\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("6 letters matched", Pattern.compile("plusieurs pe\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("7 letters matched", Pattern.compile("plusieurs p\\p{Alpha}{1,100}\\b").matcher(test).find());
assertTrue("8 letters matched", Pattern.compile("plusieurs \\p{Alpha}{1,100}\\b").matcher(test).find());

// match the negative pattern (without s or x) with curly braces
assertTrue("no letters matched, negative", Pattern.compile("plusieurs (?!personne[sx])\\w+").matcher(test).find());
assertTrue("1 letters matched, negative", Pattern.compile("plusieurs (?!personn\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
assertTrue("2 letters matched, negative", Pattern.compile("plusieurs (?!person\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
assertTrue("3 letters matched, negative", Pattern.compile("plusieurs (?!perso\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
assertTrue("4 letters matched, negative", Pattern.compile("plusieurs (?!pers\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
assertTrue("5 letters matched, negative", Pattern.compile("plusieurs (?!per\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
// the assertion below fails (is false) for reasons unknown
assertTrue("6 letters matched, negative", Pattern.compile("plusieurs (?!pe\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
assertTrue("7 letters matched, negative", Pattern.compile("plusieurs (?!p\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());
assertTrue("8 letters matched, negative", Pattern.compile("plusieurs (?!\\p{Alpha}{1,100}[sx])\\w+").matcher(test).find());

Answer 1

Let’s see how the lookahead matches:

pe           literal, matches "pe"
r            matches \p{Alpha}{1,100}
s            matches [sx]

So negative lookahead doesn’t match (the tail of your string, "onne sont", is irrelevant here).

Placing \\b after [sx] may help, if your idea is that the next word should not end with s or x. Always have in mind that the negative lookahead won’t be sorry about failing, and it won’t backtrack in order to find how it’s possible to make your regexp not match.

UPD: Let’s look closer at case 5 to compare it with case 6. Here we work with hypothetical match (for the expression inside lookahead), so we have to consider several variants of how it could (almost) happen.

per          literal, would match "per" -- it's always so
             -- let's try to imagine how the rest could match:
sonn         would match \p{Alpha}{1,100}
e            wouldn't match [sx], FAIL
             -- or maybe
s            would match \p{Alpha}{1,100}
o            wouldn't match [sx], FAIL
             -- or maybe yet
so           would match \p{Alpha}{1,100}
n            wouldn't match [sx], FAIL.

We would have another interesting adventure if the second word was “personalisation”.

UPD2: The discussion in comment prompted me to add a generalization here:

Regular expressions are attractive because they share an important trait of human mind: confirmation bias. When we write regular expressions, we want them to match; even if our job is to prevent invalid input, we think about valid inputs most of the time. Regular expression matcher normally shares this property: it wants to match and hates to fail. That’s why a subexpression like \p{Alpha}{1,100} doesn’t mean “eat the longest chunk of Alphas available, before trying to match the rest of input”. It means, roughly, “consider every possible chunk of Alphas with length within [1,100], finding a way for the whole expression to match”.

That’s why with regexps, it’s easy to overlook false positives of complex expressions: invalid inputs that are erroneously accepted. This problem doesn’t appear, it just becomes more visible when we use negative lookaheads:

Inside a negative lookahead, regexp matcher wants to match the inner expression (making outer expression fail). Human programmer still wants to match the outer expression; as we see in our example, this factor does affect our reasoning about the inner expression. We think that it shouldn’t try so hard to match (and, e.g., that it should handle subexpressions in a dumb way, instantly eating the longest input possible). The matcher works as usually, but our ideas on desirable behavior are now out of sync with its algorithm. And false positives for the inner expression (which are hard to notice), become false negatives for the outer one (which we notice and hate).