Question

I’m trying to write a simple interactive (using System.in as source) language using antlr, and I have a few problems with it. The examples I’ve found on the web are all using a per line cycle, e.g.:

while(readline)
  result = parse(line)
  doStuff(result)

But what if I’m writing something like pascal/smtp/etc, with a “first line” looks like X requirment? I know it can be checked in doStuff, but I think logically it is part of the syntax.

Or what if a command is split into multiple lines? I can try

while(readline)
  lines.add(line)
  try
    result = parse(lines)
    lines = []
    doStuff(result)
  catch
    nop

But with this I’m also hiding real errors.

Or I could reparse all lines everytime, but:

1. it will be slow
2. there are instructions I don’t want to run twice

Can this be done with ANTLR, or if not, with something else?

Answer 1

Dutow wrote:

Or I could reparse all lines everytime, but:

it will be slow
there are instructions I don’t want to run twice
Can this be done with ANTLR, or if not, with something else?

Yes, ANTLR can do this. Perhaps not out of the box, but with a bit of custom code, it sure is possible. You also don’t need to re-parse the entire token stream for it.

Let’s say you want to parse a very simple language line by line that where each line is either a program declaration, or a uses declaration, or a statement.

It should always start with a program declaration, followed by zero or more uses declarations followed by zero or more statements. uses declarations cannot come after statements and there can’t be more than one program declaration.

For simplicity, a statement is just a simple assignment: a = 4 or b = a.

An ANTLR grammar for such a language could look like this:

grammar REPL;

parse
  :  programDeclaration EOF
  |  usesDeclaration EOF
  |  statement EOF
  ;

programDeclaration
  :  PROGRAM ID
  ;

usesDeclaration
  :  USES idList
  ;

statement
  :  ID '=' (INT | ID)
  ;

idList
  :  ID (',' ID)*
  ;

PROGRAM : 'program';
USES    : 'uses';
ID      : ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '0'..'9')*;
INT     : '0'..'9'+;
SPACE   : (' ' | '\t' | '\r' | '\n') {skip();};

But, we’ll need to add a couple of checks of course. Also, by default, a parser takes a token stream in its constructor, but since we’re planning to trickle tokens in the parser line-by-line, we’ll need to create a new constructor in our parser. You can add custom members in your lexer or parser classes by putting them in a @parser::members { ... } or @lexer::members { ... } section respectively. We’ll also add a couple of boolean flags to keep track whether the program declaration has happened already and if uses declarations are allowed. Finally, we’ll add a process(String source) method which, for each new line, creates a lexer which gets fed to the parser.

All of that would look like:

@parser::members {

  boolean programDeclDone;
  boolean usesDeclAllowed;

  public REPLParser() {
    super(null);
    programDeclDone = false;
    usesDeclAllowed = true;
  }

  public void process(String source) throws Exception {
    ANTLRStringStream in = new ANTLRStringStream(source);
    REPLLexer lexer = new REPLLexer(in);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    super.setTokenStream(tokens);
    this.parse(); // the entry point of our parser
  } 
}

Now inside our grammar, we’re going to check through a couple of gated semantic predicates if we’re parsing declarations in the correct order. And after parsing a certain declaration, or statement, we’ll want to flip certain boolean flags to allow- or disallow declaration from then on. The flipping of these boolean flags is done through each rule’s @after { ... } section that gets executed (not surprisingly) after the tokens from that parser rule are matched.

Your final grammar file now looks like this (including some System.out.println‘s for debugging purposes):

grammar REPL;

@parser::members {

  boolean programDeclDone;
  boolean usesDeclAllowed;

  public REPLParser() {
    super(null);
    programDeclDone = false;
    usesDeclAllowed = true;
  }

  public void process(String source) throws Exception {
    ANTLRStringStream in = new ANTLRStringStream(source);
    REPLLexer lexer = new REPLLexer(in);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    super.setTokenStream(tokens);
    this.parse();
  } 
}

parse
  :  programDeclaration EOF
  |  {programDeclDone}? (usesDeclaration | statement) EOF
  ;

programDeclaration
@after{
  programDeclDone = true;
}
  :  {!programDeclDone}? PROGRAM ID {System.out.println("\t\t\t program <- " + $ID.text);}
  ;

usesDeclaration
  :  {usesDeclAllowed}? USES idList {System.out.println("\t\t\t uses <- " + $idList.text);}
  ;

statement
@after{
  usesDeclAllowed = false; 
}
  :  left=ID '=' right=(INT | ID) {System.out.println("\t\t\t " + $left.text + " <- " + $right.text);}
  ;

idList
  :  ID (',' ID)*
  ;

PROGRAM : 'program';
USES    : 'uses';
ID      : ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '0'..'9')*;
INT     : '0'..'9'+;
SPACE   : (' ' | '\t' | '\r' | '\n') {skip();};

which can be tested wit the following class:

import org.antlr.runtime.*;
import java.util.Scanner;

public class Main {
    public static void main(String[] args) throws Exception {
        Scanner keyboard = new Scanner(System.in);
        REPLParser parser = new REPLParser();
        while(true) {
            System.out.print("\n> ");
            String input = keyboard.nextLine();
            if(input.equals("quit")) {
                break;
            }
            parser.process(input);
        }
        System.out.println("\nBye!");
    }
}

To run this test class, do the following:

# generate a lexer and parser:
java -cp antlr-3.2.jar org.antlr.Tool REPL.g

# compile all .java source files:
javac -cp antlr-3.2.jar *.java

# run the main class on Windows:
java -cp .;antlr-3.2.jar Main 
# or on Linux/Mac:
java -cp .:antlr-3.2.jar Main

---

As you can see, you can only declare a program once:

> program A
                         program <- A

> program B
line 1:0 rule programDeclaration failed predicate: {!programDeclDone}?

---

uses cannot come after statements:

> program X
                         program <- X

> uses a,b,c
                         uses <- a,b,c

> a = 666
                         a <- 666

> uses d,e
line 1:0 rule usesDeclaration failed predicate: {usesDeclAllowed}?

---

and you must start with a program declaration:

> uses foo
line 1:0 rule parse failed predicate: {programDeclDone}?