Question

For an school project, I need to parse a text/source file containing a simplified “fake” programming language to build an AST. I’ve looked at boost::spirit, however since this is a group project and most seems reluctant to learn extra libraries, plus the lecturer/TA recommended leaning to create a simple one on C++. I thought of going that route. Is there some examples out there or ideas on how to start? I have a few attempts but not really successful yet …

parsing line by line

• Test each line with a bunch of regex (1 for procedure/function declaration), one for assignment, one for while etc…
  • But I will need to assume there are no multiple statements in one line: eg. a=b;x=1;
• When I reach a container statement, procedures, whiles etc, I will increase the indent. So all nested statements will go under this
• When I reach a } I will decrement indent

Any better ideas or suggestions? Example code I need to parse (very simplified here …)

procedure Hello {
    a = 1;
    while a {
        b = a + 1 + z; 
    }
}

Another idea was to read whole file into a string, and go top down. Match all procedures, then capture everything in { ... } then start matching statements (end with ;) or containers while { ... }. This is similar to how PEG does things? But I will need to read entire file

Answer 1

Multipass makes things easier. On a first pass, split things into tokens, like “=”, or “abababa”, or a quote-delimited string, or a block of whitespace. Don’t be destructive (keep the original data), but break things down to simple chunks, and maybe have a little struct or enum that describes what the token is (ie, whitespace, a string literal, an identifier type thing, etc).

So your sample code gets turned into:
identifier(procedure) whitespace( ) identifier(Hello) whitespace( ) operation({) whitespace(\n\t) identifier(a) whitespace( ) operation(=) whitespace( ) number(1) operation(;) whitespace(\n\t) etc.

In those tokens, you might also want to store line number and offset on the line (this will help with error message generation later).

A quick test would be to turn this back into the original text. Another quick test might be to dump out pretty-printed version in html or something (where you color whitespace to have a pink background, identifiers as light blue, operations as light green, numbers as light orange), and see if your tokenizer is making sense.

Now, your language may be whitespace insensitive. So discard the whitespace if that is the case! (C++ isn’t, because you need newlines to learn when // comments end)

(Note: a professional language parser will be as close to one-pass as possible, because it is faster. But you are a student, and your goal should be to get it to work.)

So now you have a stream of such tokens. There are a bunch of approaches at this point. You could pull out some serious parsing chops and build a CFG to parse them. (Do you know what a CFG is? LR(1)? LL(1)?)

An easier method might be to do it a bit more ad-hoc. Look for operator({) and find the matching operator(}) by counting up and down. Look for language keywords (like procedure), which then expects a name (the next token), then a block (a {). An ad-hoc parser for a really simple language may work fine.

I’ve done exactly this for a ridiculously simple language, where the parser consisted of a really simple PDA. It might work for you guys. Or it might not.