Question

With antlr, I’m trying to make a TreeWalker for a tree like this:

input: int x = 3

output AST: ^(VARDEF int x 3)

My parser works just fine and also generates an AST like shown above, but whenever I want to resolve anything from the AST, like with $variableType.text, there is allways a NullReferenceException in the generated C# 2.0 TreeWalker.

My TreeWalker:

tree grammar SGLTreeWalker;
options {
    tokenVocab = SGL;
    language = 'CSharp2'; 
}

[...]

compilationUnit
    :   (statement)+
    ;

statement
    :   variableDefinitionList
    ;

variableDefinitionList
    :   ^(VARDEF variableType variableName expression) { Console.WriteLine($variableType.text); }
    ;
[...]

The problematic part, generated by the rule “variableDefinitionList” looks like this:

Match(input, Token.UP, null);
Console.WriteLine(((variableType1 != null) ? input.TokenStream.ToString(
    input.TreeAdaptor.GetTokenStartIndex(variableType1.Start),
    input.TreeAdaptor.GetTokenStopIndex(variableType1.Start)) : null)); 

It turns out that input.TokenStream is null so it throws the NullReferenceException. I read that this can happen if the used TreeNodeStream isn’t buffered, but I used the CommonTreeNodeStream so it should be buffered I think. Here is the code I used to commit the AST:

[...]
SGLParser parser = new SGLParser(tStream);
CommonTree t = (CommonTree) parser.compilationUnit().Tree;
Console.WriteLine("; " + t.ToStringTree());
CommonTreeNodeStream  treeStream = new CommonTreeNodeStream(t);
SGLTreeWalker tw = new SGLTreeWalker(treeStream);
tw.compilationUnit();

Any idea on why input.TokenStream resolves to null when I just want to get the $variableType.text attribute?

Answer 1

That is because inside a tree grammar, the production rules return a TreeRuleReturnScope which doesn’t have a text attribute (or GetText() method).

If you want to grab the text of a variableType rule, you’ll need to explicitly return a string.

A demo:

---

SGL.g

grammar SGL;

options {
  language=CSharp2;
  output=AST;
}

tokens {
  VARDEF;
}

@parser::namespace { SGL }
@lexer::namespace { SGL }

public compilationUnit
  :  statement+ EOF
  ;

statement
  :  variableDefinitionList
  ;

variableDefinitionList
  :   variableType variableName '=' expression 
      -> ^(VARDEF variableType variableName expression)
  ;

variableType
  :  Type
  ;

variableName
  :  ID
  ;

expression
  :  Int
  ;

Type
  :  'int'
  |  'double'
  ;

Int
  :  '0'..'9'+
  ;

ID
  :  ('a'..'z' | 'A'..'Z')+
  ;

Space
  :  ' ' {Skip();}
  ;

---

SGLTreeWalker.g

tree grammar SGLTreeWalker;

options {
  tokenVocab=SGL;
  language=CSharp2; 
  ASTLabelType=CommonTree;
}

@namespace { SGL }

compilationUnit
  :  statement+
  ;

statement
  :  variableDefinitionList
  ;

variableDefinitionList
  :   ^(VARDEF variableType variableName expression) 
      {Console.WriteLine($variableType.txt);}
  ;

variableType returns [string txt]
  :  Type {$txt = $Type.text;} // note that `Type` is still a normal token! 
  ;

variableName
  :  ID
  ;

expression
  :  Int
  ;

---

And the class:

using System;
using Antlr.Runtime;
using Antlr.Runtime.Tree;

namespace SGL
{
  public class Test
  {
    public static void Main (string[] args)
    {
      ANTLRStringStream Input = new ANTLRStringStream("int x = 3");
      SGLLexer Lexer = new SGLLexer(Input);
      CommonTokenStream Tokens = new CommonTokenStream(Lexer);
      SGLParser parser = new SGLParser(Tokens);
      CommonTree t = (CommonTree) parser.compilationUnit().Tree;
      CommonTreeNodeStream  treeStream = new CommonTreeNodeStream(t);
      SGLTreeWalker tw = new SGLTreeWalker(treeStream);
      tw.compilationUnit();
    }
  }
}

produces the following output:

int