Question

With ANTLR2, you could define something like this in grammar definition file:

options
{
   language = "CSharp";
   namespace = "Extended.Tokens";
}

tokens {
   TOKEN<AST=Extended.Tokens.TokenNode>;
}

And then, you could create a class:

public class TokenNode: antlr.BaseAST
{
    ...
}

Any ideea if something like this can be used (delegate class creation to AST factory instead of me doing the tree replication manually)? It’s not working just by simple grammar definition copy from old to new format, and I tried to search their site and samples for somthing similar. Any hints?

EDIT

I’m not trying to create custom tokens, but custom ‘node parsers’.

In order to ‘execute’ a tree you have 2 choices (as far as I understood):

1. create a ‘tree visitor’ and handle values, or
2. create a tree parser by ‘almost-duplicating’ the grammar definition.

In v2 case, I could decorate the tree node to whateveer method I would have liked and then call them after the parser ran by just calling something like ‘execute’ from root node.

Answer 1

I know little C#, but there shouldn’t be much difference with the Java target.

You can create – and let ANTLR use – a custom tree by setting the ASTLabelType in the options { ... } section (an XTree in this case):

T.g

grammar T;

options {
  output=AST;
  ASTLabelType=XTree;
}

tokens {
  ROOT;
}

@parser::header {
  package demo;
  import demo.*;
}

@lexer::header {
  package demo;
  import demo.*;
}

parse
  :  Any* EOF -> ^(ROOT Any*)
  ;

Any
  :  .
  ;

You then create a custom class which extends a CommonTree:

demo/XTree.java

package demo;

import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;

public class XTree extends CommonTree {

  public XTree(Token t) {
    super(t);
  }

  public void x() {
    System.out.println("XTree.text=" + super.getText() + ", children=" + super.getChildCount());
  }
}

and when you create an instance of your TParser, you must create and set a custom TreeAdaptor which creates instances of your XTree:

demo/Main.java

package demo;

import org.antlr.runtime.*;
import org.antlr.runtime.tree.*;

public class Main {

  public static void main(String[] args) throws Exception {
    String source = "ABC";
    TLexer lexer = new TLexer(new ANTLRStringStream(source));
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.setTreeAdaptor(new CommonTreeAdaptor(){
      @Override
      public Object create(Token t) {
        return new XTree(t);
      }
    }); 
    XTree root = (XTree)parser.parse().getTree();
    root.x();
  }
}

Running the demo:

java -cp antlr-3.2.jar org.antlr.Tool T.g -o demo/
javac -cp antlr-3.2.jar demo/*.java
java -cp .:antlr-3.2.jar demo.Main

will print:

XTree.text=ROOT, children=3

For more info, see: http://www.antlr.org/wiki/display/ANTLR3/Tree+construction