Question

I’m making a lexer and a parser to parse one data format to another (partly as an exercise), and I have a question:

Let’s say we have 3 different data types, and these data types are identified by their delimiters:

1. a|b #we’ll call this type “segments”

2. a~b #we’ll call this type “array”

3. a^b #we’ll call this type “components”

You can also mix them like this:

hey~there|how~are~you

which would correspond to something like this in pseudo-code:

[["hey", "there"], ["how", "are", you"]]

and

hey~there^you~guy|hi|hehe

which would correspond to:

[[["hey", "there"], ["you", "guy"]], "hi", "hehe"]

Now my question is, in my lexer do I look ahead to see what type of data we’re dealing with so that I can emit the token type first before all the strings and delimiters get emitted? Or do I make the parser try to figure it out by the delimiter tokens it gets?

Example for hey~there^you~guy|hi|hehe:

(segment)
(component)
(array)
(string "hey")
(array_delim "~")
(string "there")
(component_delim "^")
(component)
(array)
(string "you")
(array_delim "~")
(string "guy")
(segment_delim "|")
(string "hi")
(segment_delim "|")
(string "hehe")

versus

(string "hey")
(array_delim "~")
(string "there")
(component_delim "^")
(string "you")
(array_delim "~")
(string "guy")
(segment_delim "|")
(string "hi")
(segment_delim "|")
(string "hehe")

In the first case the parser would know a component or array is coming, and make the right data structures ahead of time. In the second example it would kind of have to backtrack what it did, since it figures out what data structure it is later.

Answer 1

I believe the second example is really what you want to use. From the looks of it your syntax has the following precedence.

segments = 0
components = 1
array = 2

With that in mind you can implement an Operator-precedence parser. Which would process your tokens like this:

shift (string "hey")
shift (array_delim "~")
shift (string "there")
reduce array1
shift array1
shift (component_delim "^") 
shift (string "you")
shift (array_delim "~")
shift  (string "guy")
reduce array2
reduce component1 
shift component1 
shift (segment_delim "|")
shift (string "hi")
reduce segment1 
shift segment1 
shift (segment_delim "|") 
shift (string "hehe")
reduce segment2
done

This will result in a parse tree like this.

segment
    segment
        component
            array
                string(hey)
                string(there)
            array
                string(you)
                string(guy)
        string(hi)
    string(hehe)