Question

I’m building a parser for an asset xchange format. And I’m including the %token-table directive in the bison file but, from the flex code I just can’t access the table or the constants associated with it. That is when trying to compile this code:

Frame|FrameTransformMatrix|Mesh|MeshNormals|MeshMaterialList|Material {
    printf("A keyword: %s\n", yytext);
    yylval.charptr_type = yytext;

    int i;
    for (i = 0; i < YYNTOKENS; i++)
    {
        if (yytname[i] != 0
            && yytname[i][0] == '"'
            && !strncmp(yytname[i] + 1, yytext, strlen(yytext))
            && yytname[i][strlen(yytext) + 1] == '"'
            && yytname[i][strlen(yytext) + 2] == 0)
            return i;
    }
}

gcc says both YYNTOKENS and yytname are undeclared. So was the token table finally deprecated and wiped or what’s the deal?

Answer 1

The Bison 2.6.2 manual says (on p82 in the PDF):

%token-table [Directive]

Generate an array of token names in the parser implementation file. The name of the
array is yytname; yytname[i] is the name of the token whose internal Bison token
code number is i. The first three elements of yytname correspond to the predefined
tokens “$end“, “error“, and “$undefined“; after these come the symbols defined in
the grammar file.

The name in the table includes all the characters needed to represent the token in
Bison. For single-character literals and literal strings, this includes the surrounding
quoting characters and any escape sequences. For example, the Bison single-character
literal ’+’ corresponds to a three-character name, represented in C as "’+’"; and
the Bison two-character literal string "\\/" corresponds to a five-character name,
represented in C as "\"\\\\/\"".

When you specify %token-table, Bison also generates macro definitions for macros
YYNTOKENS, YYNNTS, and YYNRULES, and YYNSTATES:

YYNTOKENS
The highest token number, plus one.

YYNNTS The number of nonterminal symbols.

YYNRULES The number of grammar rules,

YYNSTATES
The number of parser states (see Section 5.5 [Parser States], page 104).

It looks like it is supposed to be there.

When I tried a trivial grammar, the table was present:

#if YYDEBUG || YYERROR_VERBOSE || YYTOKEN_TABLE
/* YYTNAME[SYMBOL-NUM] -- String name of the symbol SYMBOL-NUM.
   First, the terminals, then, starting at YYNTOKENS, nonterminals.  */
static const char *const yytname[] =
{
  "$end", "error", "$undefined", "ABSINTHE", "NESTLING", "$accept",
  "anything", 0
};
#endif

Notes: the table is static; if you are trying to access it from outside the file, that will not work.

There is an earlier stanza in the source:

/* Enabling the token table.  */
#ifndef YYTOKEN_TABLE
# define YYTOKEN_TABLE 1
#endif

This ensures that the token table is defined.