Question

I’m trying to create ANTLR parser for Lua. So i took grammar produced by Nicolai Mainero(available at ANTLR’s site, Lua 5.1 grammar) and begin to work.

Grammar is good. One thing not working: LONG STRINGS.

Lua specification rule: Literal

strings can also be defined using a
long format enclosed by long brackets.
We define an opening long bracket of
level n as an opening square bracket
followed by n equal signs followed by
another opening square bracket. So, an
opening long bracket of level 0 is
written as [[, an opening long bracket
of level 1 is written as [=[, and so
on. A closing long bracket is defined
similarly; for instance, a closing
long bracket of level 4 is written as
]====]. A long string starts with an
opening long bracket of any level and
ends at the first closing long bracket
of the same level. Literals in this
bracketed form can run for several
lines, do not interpret any escape
sequences, and ignore long brackets of
any other level. They can contain
anything except a closing bracket of
the proper level.the proper level.

My question is close by meaning to this but tools are different.

Some little example of LONGSTRING:

local a = [==[ Some interesting string [=[ sub string in string ]=] [hello indexes] [[And some line strings]] ]==] - its correct string. 
local f = [==[ Not interesting string ]=] - incorrect string

Here my rule for LONGSTRING with out ‘=’ symbol:

LONGSTRING: '[[' (~(']') | ']'(~(']')))* ']]';

Can somebody help me? Thanks!

Answer 1

I once wrote a Lua grammar according the specs and solved it like this:

grammar Lua;

// ... options ...

// ... tokens ...

@lexer::members {
    public boolean noCloseAhead(int numEqSigns) {
        if(input.LA(1) != ']') return true;
        for(int i = 2; i < numEqSigns+2; i++) {
            if(input.LA(i) != '=') return true;
        }
        return input.LA(numEqSigns+2) != ']';
    }

    public void matchClose(int numEqSigns) throws MismatchedTokenException {
        StringBuilder eqSigns = new StringBuilder();
        for(int i = 0; i < numEqSigns; i++) {
            eqSigns.append('=');
        }
        match("]"+eqSigns+"]");
    }
}

// ... parser rules ...

String
  :  '"'  (~('"'  | '\\') | EscapeSequence)* '"'
  |  '\'' (~('\'' | '\\') | EscapeSequence)* '\''
  |  LongBracket
  ;

Comment
  :  (BlockComment | LineComment) {skip();}
  ;

fragment
BlockComment
  :  '--' LongBracket 
  ;

fragment
LineComment
  :  '--' ~('\r' | '\n')* ('\r'? '\n' | EOF) 
  ;

fragment
LongBracket
@init {int openEq = 0;}
  :  '[' ('=' {openEq++;})* '[' ({noCloseAhead(openEq)}?=> .)* {matchClose(openEq);}
  ;

// ... more lexer rules ...

Be careful with what you find on the ANTLR Wiki! As the name suggests: it’s a Wiki and one can post stuff fairly easy. The Lua grammar you mention is a nice start, but has quite a bit of errors in it (binary or hex literals are incorrect as well, at least, at the time I looked at it…).