Home/ Questions/Q 9144511
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-17T10:17:15+00:00

I’m writing a really simple lexer for doing syntax highlighting of arbitrary text, one

  • 0

I’m writing a really simple lexer for doing syntax highlighting of arbitrary text, one of which is HTML. The goal of the lexer is just to provide a flat stream of tokens.

I started with the XML tutorial on the Antlr3 website, but am having some trouble with script tags.

An example of the HTML which causes this problem:

<head>
<script>alert(2 < 3);</script>
</head>

And the grammar..

@members {
    boolean inTag = false;
}

TAG_START_OPEN : '<'
                 { inTag = true; } ;
TAG_END_OPEN : '</'
               { inTag = true; } ;

TAG_CLOSE : { inTag }?=> '>' { inTag = false; } ;
TAG_SELF_CLOSE : { inTag }?=> '/>' { inTag = false; } ;
PCDATA : { !inTag }?=> (~'<')+ ;

// ...

The problem is that the lexer gets confused when seeing the ‘<‘ tag within the Javascript code and thinks it is a close tag. I guess the goal would be for the lexer to use lookahead to determine whether a ‘<‘ is proceeded by ‘/script>’ if the open tag was a script tag, however I’m unsure of how to do this nicely with ANTLR.

Thanks in advance for any help.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Here’s a quick demo of how you could accomplish this:

    grammar T;

options {
  output=AST;
}

tokens {
  DATA;
  ATTRIBUTES;
  ATTRIBUTE;
  ATOMS;
}

@lexer::members {
  private boolean inTag = false;
  private boolean isScript = false;

  private boolean ahead(String s) {
    for(int i = 0; i < s.length(); i++) {
      int ch = input.LA(i + 1);
      if(ch != s.charAt(i)) {
        return false;
      }
    }
    return true;
  }
}

parse
 : tag EOF -> tag
 ;

tag
 : TagOpen attributes TagOpenEnd atoms TagClose -> ^(TagOpen attributes atoms)
 ;

attributes
 : attribute* -> ^(ATTRIBUTES attribute*)
 ;

attribute
 : Key Assign Value -> ^(ATTRIBUTE Key Value)
 ;

atoms
 : atom* -> ^(ATOMS atom*)
 ;

atom
 : PCData
 | ScriptData
 | tag
 ;

TagOpen
 : '<' Name 
   {
     inTag=true; 
     isScript = $Name.text.equals("script");
     setText($Name.text);
   }
 ;

TagClose
 : {!inTag}?=> '</' Name '>' 
   {
     isScript = false;
     setText($Name.text);
   }
 ;

TagOpenEnd
 : {inTag}?=> '>' {inTag=false;}
 ;

Key
 : {inTag}?=> Name
 ;

Assign
 : {inTag}?=> '='
 ;

Value
 : {inTag}?=> '"' ~'"'* '"'
   {
     setText($text.substring(1, $text.length() - 1));
   }
 ;

PCData
 : {!inTag && !isScript}?=> ~'<'+
   {
     if($text.trim().isEmpty()) {
       skip();
     }
   }
 ;

ScriptData
 : {!inTag && isScript}?=> ({!ahead("</script>")}?=> . )+
 ;

Space
 : {inTag}?=> (' ' | '\t' | '\r' | '\n')+ {skip();}
 ;

fragment Name : ('a'..'z' | 'A'..'Z')+;

    If I now parse the input:

    <head>
  <script> alert(2 < 3); </script> 
  <span key="some value" x="<>">
    Mu <em>foo</em> bar!
  </span>
</head>

    the following AST will be created by the generated parser:

    enter image description here

    • 0