Question

I’ve just read here (http://ruby.runpaint.org/programs#lexical) that comments are tokens. I’ve never thought of comments as tokens as they’re either annotations or for a post-processor.

Are comments really tokens or is this source wrong?

Answer 1

Yeah they’re tokens to the parser. Usually, if you use a parser generator this is the definition of a comment

{code} short_comment = '//' not_cr_lf* eol | '#' not_cr_lf* eol;
{code} long_comment = '/*' not_star* '*'+ (not_star_slash not_star* '*'+)* '/';  /* '4vim */
Ignored Tokens
  short_comment,
  long_comment;

This is a SableCC grammar. They’re usually ignored tokens.

Remember that everything you write in a source code is a token, that’s always the first step. The parser needs to start building the abstract syntax tree from tokens.