Question

In ANTLR, I have a MismatchedTokenException with the following definition:

type : IDENTIFIER ('<' (type (',' type)*) '>')?;

And the following test:

A<B,C<D>>

The exception occurs when parsing the first >. ANTLR tries parsing both ‘>>’ at once, and fails.

With a silent whitespace channel, the following test does work:

A<B,C<D> >

In which ANTLR is clearly instructed to treat each token separately.

How can I fix that?

Answer 1

I could not reproduce that. The parser generated by:

grammar T;

type       : IDENTIFIER ('<' (type (',' type)*) '>')?;
IDENTIFIER : 'A'..'Z';

parses the input A<B,C<D>> (without spaces) into the following parse tree:

You’ll need to provide the grammar that causes this input to produce a MismatchedTokenException.

Perhaps you’re using ANTLRWorks’ interpreter (or Eclipse’s ANTLR-IDE, which uses the same interpreter)? In that case, that is probably the problem: it’s notoriously buggy. Don’t use it, but use ANTLRWorks’ debugger: it’s great (the image posted above comes from the debugger).

Lazlo Bonin wrote:

Got it. I had a << token defined. Quickly, is there a way to priorize token recognition over another?

No, the lexer simply tries to match as much as possible. So if it can create a token matching << (or >>), it will do so in favor of two single < (or >) tokens. Only when two (or more) lexer rules match the same amount of characters, a prioritization is made: the rule defined first will then “win” over the one(s) defined later in the grammar.