Question

How do you do something like this with ANTLR?

Example input:

title: hello world

Grammar:

header : IDENT ':' REST_OF_LINE ;
IDENT : 'a'..'z'+ ;
REST_OF_LINE : ~'\n'* '\n' ;

It fails, with line 1:0 mismatched input 'title: hello world\n' expecting IDENT

(I know ANTLR is overkill for parsing MIME-like headers, but this is just at the top of a more complex file.)

Answer 1

It fails, with line 1:0 mismatched input ‘title: hello world\n’ expecting IDENT

You must understand that the lexer operates independently from the parser. No matter what the parser would “like” to match at a certain time, the lexer simply creates tokens following some strict rules:

1. try to match tokens from top to bottom in the lexer rules (rules defined first are tried first);
2. match as much text as possible. In case 2 rules match the same amount of text, the rule defined first will be matched.

Because of rule 2, your REST_OF_LINE will always “win” from the IDENT rule. The only time an IDENT token will be created is when there’s no more \n at the end. That is what’s going wrong with your grammars: the error messages states that it expects a IDENT token, which isn’t found (but a REST_OF_LINE token is produced).

I know ANTLR is overkill for parsing MIME-like headers, but this is just at the top of a more complex file.

You can’t just define tokens (lexer rules) you want to apply to the header of a file. These tokens will also apply to the rest of the more complex file. Perhaps you should pre-process the header separately from the rest of the file?