Question

In compiler construction, when you talk about tokens, is a token the same like a symbol / just another term for a symbol? After some research I think to understand, that a token is a symbol with a reference to the symbol table, therefore some kind of attributed symbol / a symbol with some additional informations? Thanks for any clearfication 🙂

Answer 1

A token is not necessarily a symbol in the symbol table.
For example, if a token is a reserved word, then it is not entered in the symbol table. If a token is an identifier, then it will likely be entered in the symbol table.

Take for example the following declaration:

char s[100];

A lexical analyzer could output the following tokens:

<"char", IDENTIFIER>

depending on the implementation it could be recognized as a reserved word or be entered in the symbol table as a predefined type name (I am not 100% sure here),

<"s", IDENTIFIER>

“s” is entered in symbol table as a variable identifier,

<"[", OPEN_SQUARE_BRACKET>

not entered in symbol table,

<"100", INTEGER_LITERAL>

not entered in symbol table,

<"]", CLOSE_SQUARE_BRACKET>

not entered in symbol table,

<";", SEMI_COLON>

not entered in symbol table.

So you basically enter in the symbol table only those tokens that you need to reference later during the compilation process. E.g., later in the function body, when you find

strcpy(s, "Hello, world\n");

you recognize again the token <“s”, IDENTIFIER> and look it up in the symbol table. The symbol table will say that “s” has been declared as a variable of type char [].

So , I would say a token is any chunk of input that is recognized by the lexical analizer, and that only certain tokens with a special meaning are entered as symbols in the symbol table.