Question

I’m working with a (Python-flavored) regular expression to recognize common and idiosyncratic forms and abbreviations of scripture references. Given the following verbose snippet:

>>> cp = re.compile(ur"""
    (?:(
        # Numbered books
        (?:(?:Third|Thir|Thi|III|3rd|Th|3)\ ? 
           (?:John|Joh|Jhn|Jo|Jn|Jn|J))
        # Other books
        |Thessalonians|John|Th|Jn)\ ? 
      # Lookahead for numbers or punctuation
      (?=[\d:., ]))

    |

    # Do the same check, this time at the end of the string.
    (
      (?:(?:Third|Thir|Thi|III|3rd|Th|3)\ ?
         (?:John|Joh|Jhn|Jo|Jn|Jn|J))
      |Thessalonians|John|Th|Jn)\.?$
    """, re.IGNORECASE | re.VERBOSE)

>>> cp.match("Third John").group()
'Third John'

>>> cp.match("Th Jn").group()
'Th'

>>> cp.match("Th Jn ").group()
'Th Jn'

The intention of this snippet is to match various forms of “Third John”, as well as forms of “Thessalonians” and “John” by themselves. In most cases this works fine, but it does not match “Th Jn” (or “Th John”), rather matching “Th” by itself.

I’ve ordered the appearance of each abbreviation in the expression from longest to shortest expressly to avoid a situation like this, relying on a regular expression’s typically greedy behavior. But the positive lookahead assertion seems to be short-circuiting this order, picking the shortest match instead of the greediest match.

Of course, removing the lookahead assertion makes this case work, but breaks a bunch of other tests. How might I go about fixing this?

Answer 1

I’ve given up after a little try to follow what _sre.so is doing in this case (too complicated!) but a “blind fix” I tried seemed to work — switch to a negative lookahead assertion for the complementary character set…:

cp = re.compile(ur"""
(?:(
    # Numbered books
    (?:(?:Third|Thir|Thi|III|3rd|Th|3)\ ? 
       (?:John|Joh|Jhn|Jo|Jn|Jn|J))
    # Other books
    |Thessalonians|John|Th|Jn)\ ? 
  # Lookahead for numbers or punctuation
  (?![^\d:., ]))

|

etc. I.e. I changed the original (?=[\d:., ])) positive lookahead into a “double negation” form (negative lookahead for complement) (?![^\d:., ])) and this seems to remove the perturbation. Does this work correctly for you?

I think it’s an implementation anomaly in this corner case of _sre.so — it might be interesting to see what other RE engines do in these two cases, just as a sanity check.