Question

I’m using ANTLR 2.7.6 to parse the messy output of another application. Sadly, I do not have the ability to upgrade to ANTLR 3, even though it has been out for quite a while. A log file of the sort I will be parsing is better conceptualized as a list of objects than a tree of objects, and could be very large (>100 MB) so it is not practical to read it all into one AST. (My application is multithreaded and will process half a dozen to a dozen of these files at once, so memory will fill up quick.) I want to be able to read out each of these objects as from a stream so I can process them one by one. Note that the objects themselves could be conceptualized as small trees. Is there a way to get my ANTLR parser to act like an object stream, an iterator, or something of that nature?

[See Javadoc for ANTLR 2.]

Edit: Here is a conceptual example of what I would like to do with the parser.

import java.io.FileReader;
import antlr.TokenStream;
import antlr.CharBuffer;
//...
FileReader fileReader = new FileReader(filepath);
TokenStream lexer = new MyExampleLexer(new CharBuffer(fileReader));
MyExampleParser parser = new MyExampleParser(lexer);
for (Object obj : parser)
{
    processObject(obj);
}

Am I perhaps working with the wrong paradigm of how to use an Antlr parser? (I realize that the parser does not implement Iterator; but that is conceptually the sort of behavior I’m looking for.)

Answer 1

AFAIK, ANTLR v2.x buffers the creating of tokens. The parser takes a TokenBuffer, which in its turn takes a TokenStream. This TokenStream is then polled through its nextToken() method when the parser needs more tokens.

In other words, if you provide the input source as a file, ANTLR does not read the entire file and create tokens of it, but only when needed are tokens created (and discarded).

Note that I never worked with ANTLR 2.x, so I could be wrong. Have you observed something different? If so, how do you offer the source to ANTLR: as a file, or as a big string? If it’s the latter, I recommend providing a file instead.

EDIT

Let’s say you want to parse a file that consists of lines with numbers, delimited by white spaces (which you want to ignore). You also want your parser to process the file line by line because collecting all numbers at once would result in memory problems.

You can do this by letting your main parser rule, parse, return a list of numbers for each line. If the EOF (end-of-file) is reached, you simply return null instead of a list.

A demo using ANTLR 2.7.6:

file: My.g

class MyParser extends Parser;

parse returns [java.util.List<Integer> numbers]
{
  numbers = new java.util.ArrayList<Integer>();
}
  :  (n:Number {numbers.add(Integer.valueOf(n.getText()));})+ LineBreak
  |  EOF {numbers = null;}
  ;

class MyLexer extends Lexer; 

Number
  :  ('0'..'9')+
  ;

LineBreak
  :  ('\r')? '\n'
  ;

Space
  :  (' ' | '\t') {$setType(Token.SKIP);}
  ;

file: Main.java

import antlr.*;

public class Main {
  public static void main(String[] args) throws Exception {
    MyLexer lexer = new MyLexer(new java.io.StringReader("1 2 3\n4 5 6 7 8\n9 10\n"));
    MyParser parser = new MyParser(new TokenBuffer(lexer));
    int line = 0;
    java.util.List<Integer> numbers = null;
    while((numbers = parser.parse()) != null) {
      line++;
      System.out.println("line " + line + " = " + numbers);
    }
  }
}

To run the demo on:

*nix

java -cp antlr-2.7.6.jar antlr.Tool My.g
javac -cp antlr-2.7.6.jar *.java
java -cp .:antlr-2.7.6.jar Main

or on:

Windows

java -cp antlr-2.7.6.jar antlr.Tool My.g
javac -cp antlr-2.7.6.jar *.java
java -cp .;antlr-2.7.6.jar Main

which will produce the following output:

line 1 = [1, 2, 3]
line 2 = [4, 5, 6, 7, 8]
line 3 = [9, 10]

---

Warning

Anyone trying this code, please note that this uses ANTLR 2.7.6. Unless you have a very compelling reason to use this version, it is highly recommended to use the latest stable version of ANTLR (v3.3 at the time of this writing).