Question

I have created a lexer using antlr 4 for tokenizing Turkish natural language texts, what I need to do is to have a token stream which I can fetch tokens one by one. CommonTokenStream returns a List if I use it like this:

ANTLRInputStream inputStream = new ANTLRInputStream(input);
TurkishLexer lexer = new TurkishLexer(inputStream);
CommonTokenStream tokenStream = new CommonTokenStream(lexer);
List<Token> tokens = tokenStream.fill();
for (Token token : token) ...

However I don’t want to construct a list of tokens as my input could be huge, I just want something like:

for (Token token: tokenStream.next()) ...

Which I would iterate until getting an EOF token.

Is there a Token Stream that allows me to iterate over tokens?

Answer 1

Rather than use a CommonTokenStream, you could simply use Lexer.nextToken.

for (Token token = lexer.nextToken();
     token.getType() != Token.EOF;
     token = lexer.nextToken())
{
    ...