I just started reading The Dragon Book and I’m finding difficulty in understanding some statements.
It says: “lexical analyser produce sequence of token for each lexeme in the source program”. Can you please help me to understand the above line? I know about tokens and lexemes, but what is meant by producing multiple token for each lexeme….AFAIK LEXEME itself compromising a single token.
The complete quote is as follows:
“As the first phase of compiler, the main task of the Lexical Analyser is to read the input characters of the source program, group them into lexemes, and produce as output a sequence of tokens for each lexeme in the source program.”
The above quote is from chapter 3..section 3.1 under the heading “role of lexical analyser” page number is 109
You are correct. Tokens usually correspond one-to-one with lexemes. Try re-parsing that sentence as the “…and produce as output a sequence of tokens for the lexemes in the source program.” That is the meaning the authors intended, as I read it.