Question

Assuming that i need to tokenize and parse only multiline comments, how will i do that using Parse::Lex. When using flex-bison, the default action for any pattern in the rules section of the lex file used to be ‘skip’.

%%
.*    ;
%%

How to do this here ?

[EDIT] Well, i tried that, i’m still missing something – here is my code – and result. Where have i gone wrong ??

my simplified lex file:

use Parse::Lex;
use Regexp::Common;
use YParser;
my $lexer;
my @token = (
qw|esp:TA abcdefgh|,
qw(esp:REST .|\n),
);
Parse::Lex->trace;
Parse::Lex->exclusive('esp');
$lexer = Parse::Lex->new(@token);
$lexer->from(\*STDIN);
$lexer->skip(qr! [ \t]+ | $RE{balanced}{-begin=>'/*'}{-end=>'*/'} !xms);
$lexer->start('esp');

my $j = YParser->new();
$j->YYParse(yylex => \&lex);

sub lex {
    my $token = $lexer->next;
    return ('', undef) if $lexer->eoi;
    if ($token->name eq 'TA' || $token->name eq 'REST') {
        return ($token->name, {LINENO => $lexer->line, TEXT => $token->text});
    }
}

my simplified grammar file

% token TA REST

%%

Program:  Element
          | Program Element
;
Element:  TA
          | REST
;

%%

Input file:

abcdefgh
/*sdf*/

Result:
perl lexfile.pl < inputfile

Trace is ON in class Parse::Lex
Can't call method "name" on an undefined value at qnlex.pl line 26, <STDIN> line 1.

Answer 1

Well, i figured this out 🙂 Very simple – all i have to do is make the lex get the next token when encountering tokens i want to skip. Below is code to skip passing the token ‘REST’ to the parser.

sub lex {
    my $token;
    NEXTTOKEN:
    $token = $lexer->next;
    return ('', undef) if $lexer->eoi;
    if ($token->name eq 'TA') {
        return ($token->name, {LINENO => $lexer->line, TEXT => $token->text});
    }
    elsif ($token->name eq 'REST') {
        goto NEXTTOKEN;
    }
}