Question

I am implementing a compiler for a school class and I have some problems on how to proceed.
Lexical analysis is already made. Lexical analysis to me is that I have a function that returns token objects and also prints some errors that can be inspected during lexical analysis.

The token holds id, string (if the current id is a string, if not it is null), number (if the token is a number, if not it is null) and the line where the token found.

I want to make the syntax analysis but i am not sure if i have to generate a syntax tree during this procedure. I am sure that this will be necessary while generating intermediate code but the teacher leaves the decision to us.

So to end. The teacher want us to understand that it is necessary?
If it is really necessary what is the best way to construct a syntax tree?
Also am i missing something that will cause me trouble in later stages?

Answer 1

Basically, your syntax analysis will end up being some form of a finite-state machine. The result of this process is usually an AST; the syntax analysis seems somewhat pointless if you don’t store the result of it somewhere.

While there are many different well-known and established algorithms for creating the state tables and for implementing the actual processor, you might want to start thinking like the compiler and define your states by hand (which is feasible for really simple languages):

• At the very beginning, what tokens would be acceptable? (Start state)
• Each token will lead you to another state and possibly have you perform an operation to consolidate your tokens into an abstract syntax tree. (State transitions)
• I suggest to treat “End of file” as a special token returned by the tokenizer, so that your syntax analysis doesn’t need any special code to handle the end of file (just normal state transition dealing with the EOF token).

Note that, instead of using tables, you can also use functions to represent your states.

What is the language you’re trying to implement?