Home/ Questions/Q 6972763
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T17:00:46+00:00

I’m using the Whittle gem to parse a template language and wanted to match

  • 0

I’m using the Whittle gem to parse a template language and wanted to match anything that is not contained in a rule. I am well aware of other template engines out there but this is more of an academic exercise than a production case.

The problem I encounter is that the parser ignores the priority of :id above :raw and still awaits for a :raw tag after the {{.

How to tell the parses that it’s not allowed to apply the :raw rule inside an expression and only apply the :spc rule inside of an expression ?

Parser code

class Parser < Whittle::Parser
    # Skip whitespaces (should not apply in :raw)
    rule(:spc => /\s+/).skip!

    # Various delimiters
    rule("{{") ^ 4
    rule("}}") ^ 4
    rule("{%") ^ 4
    rule("%}") ^ 4
    rule("|") ^ 4
    rule("end") ^ 4

    # Defines an id (very large match)
    rule(:id => /[a-zA-Z_.$<>=!:]+(\((\w+|\s+|,|")+\))?/) ^ 2

    # inline tag
    rule(:inline) do |r|
        r["{{", :inline_head, "}}"].as { |_,id,_| Tag::Inline.new(id) }
    end
    # inline tag contents
    # allows "|" chaining
    rule(:inline_head) do |r|
        r[:inline_head, "|", :id].as { |head, _, id| head << id }
        r[:id].as { |id| [id] }
        r[].as { [] }
    end

    # block tag
    rule(:block) do |r|
        r["{%", :block_head, "%}", :all, "{%", "end", "%}"].as { |_,head,_,tags,_,_,_|
            Tag::Block.new(head, tags)
        }
    end
    # block tag heading
    # separates all the keywords
    rule(:block_head) do |r|
        r[:block_head, :id].as { |head, id| head << id }
        #r[:id].as { |id| [id] }
        r[].as { [] }
    end

    # one rule to match them all
    rule(:all) do |r|
        r[:all,:inline].as { |all, inline| all << inline }
        r[:all, :block].as { |all, block| all << block }
        r[:all, :raw].as { |all, raw| all << raw }
        r[].as { [] }
    end

    # the everything but tags rule
    rule(:raw => /[^\{\}%]+/).as { |text| Tag::Raw.new(text) } ^ 1

    # starting rule
    start(:all)
end

And the input text would be and the output is an abstract syntax tree represented by objects (they are simply hash like objects for now).

<html>
    <head>
        <title>{{ title|capitalize }}</title>
    </head>
    <body>
        <div class="news">
            {% for news in articles %}
                {{ news.title }}
                {{ news.body | limit(100) }}
                {{ tags | join(",", name) }}
            {% end %}
        </div>
    </body>
</html>
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I don’t believe the operator precedence support plays a role here. Operator precedences only come into play when resolving ambiguities in expressions like foo = 6 + 7, where the expression could either be interpreted as (foo = 6) + 7 or foo = (6 + 7). Giving non-operators a precedence doesn’t really serve any purpose.

    Perhaps it’s not clear what the parser actually does. It basically loops repeated, matching all terminal rules against your input string. For the ones it finds, it takes the longest one and tries to find a rule in the current state that it will fit into. So the parser will always find your whitespace and discard it, since that is the first rule in your grammar.

    I think you actually don’t want to skip whitespace, since it is significant in your grammar. You want to include it in your rules that allow it; which will make your grammar more verbose, but is (currently) unavoidable.

    So :raw becomes something like the following, swallowing up all whitespace and non-syntax tokens into a single string:

    rule(:raw => /[^\s\{\}%]+/)

rule(:text) do |r|
  r[:text, :raw].as { |text, raw| text << raw }
  r[:text, :spc].as { |text, spc| text << spc }
  r[:spc]
  r[:raw]
end

    Then in your :all rule, turn that text into part of your AST (you could actually do this in the above rule too, but I don’t know anything about your class definitions).

    rule(:all) do |r|
  # ... snip ...
  r[:all, :text].as { |all, text| all << Tag::Raw.new(text) }
  # ... snip ...
end

    I’ve been thinking about how to provide the ability to match different tokens within different states, but I’m wary of writing a clone of lex/flex, which I think would be too confusing, so I’m trying to come up with an approach that uses blocks to nest rules inside each other to convey how states relate to each other; though it’s not simple to create an easy to comprehend DSL that does this 😉 I also want to provide an optional DSL to hide the algorithm used for repetition; maybe providing a sort of PEG layer that transforms into a LR parser. This is still a (very) young project 😉

    • 0