Question

I’m an amateur with ANTLR and I’m creating an interpreter for a simple processor and I have run into a small issue with the VALUE token throwing errors. I’m a student, so I’m not asking you to do my homework for me…I’ve pretty much completed it (including all class files for the interpreter) but this one issue is defeating me even though it’s likely simple and staring me right in the face.

ANTLR works keeps giving me this console error message;

“error(208): newExpr.g:193:1: The following token definitions can never be matched because prior tokens match the same input: VALUE”

Clearly there is something wrong with the regular expressions for VALUE but I can’t see what it is, either there or anywhere else in the grammar. If you could point out what I’m missing it would be much appreciated…as googling hasn’t really helped me find the error in my own grammar.

grammar newExpr;

options 
{
    language=Java;
}

@header 
{
    import java.util.*;
}

@members 
{
    ArrayList myInitialise = new ArrayList();
    ArrayList InstructionList = new ArrayList();
}

/*--------------------------------------------------------------------------------------------------------------------------------*
 * PARSER RULES                                                                                                                   *
 *--------------------------------------------------------------------------------------------------------------------------------*//

/*
* prog is where the interpretation beings and consists of one or more (+) 'stat' rules
*/
prog        :       stat+;

/*
* stat rules are the general parse rules of entire operations on the processor.
* They consist of smaller data operations rules (dataop) or memory operations (memop).
*/                
stat        :       BASIC r1=REG c1=COMMA r2=REG c2=COMMA dataop NEWLINE
            {
                int reg1 = Integer.parseInt($r1.text.substring(1));  // these lines convert the token input stream and converts to an actual integer
                int reg2 = Integer.parseInt($r2.text.substring(1)); 
                int IMDT = $dataop.value;    // take the immediate integer

                // LOAD operation
                if($BASIC.text.equals("LD"))
                InstructionList.add(new ld(reg1, reg2, IMDT));

                // STORE operation  
                else if($BASIC.text.equals("ST"))
                InstructionList.add(new st(reg1, reg2, IMDT));

                // SUBTRACTION operation    
                else if($BASIC.text.equals("SUB"))
                InstructionList.add(new sub(reg1, reg2, IMDT));

                // ADDITION operation   
                else if($BASIC.text.equals("ADD"))
                InstructionList.add(new add(reg1, reg2, IMDT));

                // MULTIPLICATION operation 
                else if($BASIC.text.equals("MUL"))
                InstructionList.add(new mul(reg1, reg2, IMDT));

                // DIVISION operation   
                else if($BASIC.text.equals("DIV"))
                InstructionList.add(new div(reg1, reg2, IMDT));
            }

            | 

            i1 = INDEX '=' memop NEWLINE
            {
                myInitialise.add(new memInit(Integer.parseInt($i1.text), $dataop.value));
            }

            |

            JUMP REG COMMA dataop NEWLINE
            {
                int R = Integer.parseInt($REG.text.substring(1));
                int val = $dataop.value;

                // BRANCH EQUAL operation
                if($JUMP.text.equals("BEZ"))
                InstructionList.add(new branchEqualZero(R,value));

                // BRANCH NOT EQUAL operation
                else if($JUMP.text.equals("BNEZ"))
                InstructionList.add(new branchNotEqualZero(R,value));
            }

            | 

            JUMP REG NEWLINE
            {
                int R = Integer.parseInt($REG.text.substring(1));
                InstructionList.add(new jump(R));
            }

            | 

            HALT 
            {
                InstructionList.add(new halt());
            }
            ;


dataop returns [int value] 

        :   INDEX
            {
                $value = Integer.parseInt($INDEX.text);
            }

            |   

            VALUE
            {
                $value = Integer.parseInt($VALUE.text.substring(1))*-1;
            };


memop returns [int value]

        :   INDEX
            {
                $value = Integer.parseInt($INDEX.text);
            }

            |

            VALUE
            {
                $value = Integer.parseInt($VALUE.text.substring(1))*-1;
            }

            |

            MEMVAL
            {
                if($MEMVAL.text.startsWith("-"))
                {
                    $value = Integer.parseInt($MEMVAL.text.substring(1))*-1;
                }
                else
                    $value = Integer.parseInt($MEMVAL.text);
            };


/*--------------------------------------------------------------------------------------------------------------------------------*
 * LEXER RULES                                                                                                                    *
 *--------------------------------------------------------------------------------------------------------------------------------*/

/*
* RegExps for BASIC instructions (load, store, add, subtract, multiply, divide
*/
BASIC       :   ('L' 'D') | ('S' 'T') | ('A' 'D' 'D') | ('S' 'U' 'B') | ('M' 'U' 'L') | ('D' 'I' 'V');

/*
* The comma is simply for syntactic purposes, to separate data and register references
*/
COMMA       :   ',';

/*
* Regular Expressions for the processor registers R0-R31
*/
REG         :   ('R') (('0'..'9') | ('0'..'2') ('0'..'9') | ('3') ('0'..'1') );

/*
* 'Index' is the set of regular expressions matching memory locations
*/
INDEX       :       ('0'..'9')                    
            |
            ('0'..'9') ('0'..'9')
            |
            ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('0'..'9') ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('0'..'5') ('0'..'9') ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('6') ('0'..'4') ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('6') ('5') ('0'..'4') ('0'..'9') ('0'..'9')
            |
            ('6') ('5') ('5') ('0'..'2') ('0'..'9')
            |
            ('6') ('5') ('5') ('3') ('0'..'5');

/*
* Reg Exps for memory initialisation instructions
*/
MEMVAL      :   ('0'..'9')+ | '-' ('0'..'9')+;            

/*
* Simple integers for data values
*/          
VALUE       :   '-' (('0'..'9')         **PROBLEM IS HERE**
            |
            ('0'..'9') ('0'..'9')
            |
            ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('0'..'9') ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('0'..'5') ('0'..'9') ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('6') ('0'..'4') ('0'..'9') ('0'..'9') ('0'..'9')
            |
            ('6') ('5') ('0'..'4') ('0'..'9') ('0'..'9')
            |
            ('6') ('5') ('5') ('0'..'2') ('0'..'9')
            |
            ('6') ('5') ('5') ('3') ('0'..'6'));

/*
* Regular Expressions for return/newline characters
*/ 
NEWLINE     :   '\r'? '\n' ;


/*
* This simply makes the interpreter tolerant to whitespace
*/
WHITESPACE      :   (' ' | '\t' | '\u000C')+ {skip();};

/*
* RegExp for Branch on Equal to Zero/Branch on Not Equal to Zero instructions
*/
BRANCH      :   ('B' 'E' 'Z') | ('B' 'N' 'E' 'Z');

/*
* RegExp for jump instruction
*/
JUMP        :   ('J' 'R');

/*
* The HALT instruction ends the program and executes all instructions
* in the Instruction List on the data/values that have been entered
*/
HALT        :   ('H' 'A' 'L' 'T');

Answer 1

ANTLR’s generated lexer works like this: it tries to match as much as possible, and when two (or more) rules match the same amount of characters, the rule defined first will “win”. Because of this, your VALUE rule can never “win” from the MEMVAL rule since everything that is matched by VALUE is also matched by MEMVAL‘s: '-' ('0'..'9')+.

Hence the error message you see.

It does not matter if one of your parser rules might need a VALUE token at a certain moment, the lexer will simply produce a token based on the rules I mentioned: the lexer does not take any information from the parser into account.

Simply remove the VALUE rule and replace it with MEMVAL (and perhaps rename MEMVAL to INT). Then in your parser rule simply match MEMVAL (or INT) and check if this value is within a specific numerical range.