Question

I’m working on a parser for a simple language using JavaCC.

In my token declaration section, I have something like this

< CONSTANT: < INTEGER > | < FLOAT_NUMBER > | < BOOLEAN > >

< INTEGER: "0" | ("-")? ["1"-"9"](["0"-"9"])* | "0x" (["0"-"9"] | ["a"-"f"]  | ["A"-"F"])+ >
< FLOAT_NUMBER: ("-")? (["0"-"9"])+ "." (["0"-"9"])+ ("E" ("-")? (["0"-"9"])+)? ("D")? >
< BOOLEAN: "true" | "false">

CONSTANT, due to its order, is of higher priority than INTEGER. There are, however, parts of the grammar where I need an INTEGER, specifically. In these cases, the parser throws an exception because the INTEGER was parsed as a CONSTANT.

A simple solution is to eliminate the CONSTANT token and find-and-replace it for (< INTEGER > | < FLOAT_NUMBER > | < BOOLEAN >). This feels rather clunky, though.

Is there a way to define CONSTANT as simply a grouping of other tokens, such that CONSTANT is not a token by itself but shorthand for (< INTEGER > | < FLOAT_NUMBER > | < BOOLEAN >)?

Thank you for your time.

Answer 1

I think the simple answer here is that you shouldn’t be doing this in the tokenizer.

It is better to put this into a production rule such as :

Token parseConstant() :
{
    Token t;    
}

{
    (
      t = < INTEGER > | t = < FLOAT_NUMBER > | t = < BOOLEAN >
    )
    {
      return t;
    }

}