Question

I am trying to learn Lex. I have a simple program where i want to read in a file and recognize tokens.

Right now i am getting some errors. I think i am having problems because there is more than one line in the file to recognize the tokens?

Here is the file

fd 3x00
bk
setc 100
int xy3 fd 10 rt 90

here is the output i am trying to achieve:

Keyword: fd
Illegal: 3x00
Keyword: bk
Keyword: setc
Number: 100
Keyword: int

here is my program:

%{

/* Comment  */

 #include <stdio.h>
 #include <stdlib.h>

%}
%%
fd                     {printf("Keyword: fd\n");}
[0-9][a-z][0-9]        {printf("Illegal: 3x00\n");}
bk                     {printf("Keyword: bk\n");}
setc[0-9]              {printf("Keyword: setc\n Number: %s\n", yytext);}
int                    {printf("Keyword: int\n");}
xy3                    {printf("ID: xy3\n");}
fd[0-9]                {printf("Keyword: fd\n Number %s\n", yytext);}
rt[0-9]                {printf("Keyword: rt \n Number %s\n", yytext);}
%%

main( argc, argv)
int argc;
char** argv;
{
    if(argc > 1)
    {

        FILE *file;
        file = fopen(argv[1], "r");
        if(!file)
        {
           fprintf(stderr, "Could not open %s \n", argv[1]);
           exit(1);
        }

        yyin = file;

    }

    yylex();

}

here are the errors i am getting when i try to compile it:

 In function 'yylex':
miniStarLogo.l:11: error: expected expression before '[' token
miniStarLogo.l:11: error: 'a' undeclared (first use in this function)
miniStarLogo.l:11: error: (Each undeclared identifier is reported only once
miniStarLogo.l:11: error: for each function it appears in.)
miniStarLogo.l:11: error: expected ';' before '{' token
miniStarLogo.l:13: error: expected expression before '[' token
miniStarLogo.l:13: error: expected ';' before '{' token

is the error in my printf statements?
Thank you

Answer 1

When I compiled a copy of your code on MacOS X (10.7.2) with flex (2.5.35) and gcc (4.6.1), the only complaints I got from the C compiler were about the non-prototype definition of main(), and that was because I always compile with that warning enabled and mention of yyunput() defined but not used (which is not your fault).

Since you’re learning C, you should only be using the notation:

int main(int argc, char **argv)
{
    ...
}

or an equivalent.

I also converted the miniStarLogo.l file to DOS format (CRLF line endings), and both flex and gcc seemed to be OK with the results – somewhat to my surprise. It might not be the case on your machine.

When I ran the code on your test data, I got:

Keyword: fd
 Illegal: 3x00
0
Keyword: bk

setc 100
Keyword: int
 ID: xy3
 Keyword: fd
 10 rt 90

So, you are not far off where you need to be by my reckoning.

---

Confusion reigneth over my commands.

I used (hmmm, it was GCC 4.2.1 rather than 4.6.1), but:

$ flex miniStarLogo.l
$ gcc -Wall -Wextra -O3 -g -o lex.yy lex.yy.c -lfl
miniStarLogo.l:22: warning: return type defaults to ‘int’
miniStarLogo.l: In function ‘main’:
miniStarLogo.l:42: warning: control reaches end of non-void function
miniStarLogo.l: At top level:
lex.yy.c:1114: warning: ‘yyunput’ defined but not used
$ ./lex.yy <<EOF
> fd 3x00
> bk
> setc 100
> int xy3 fd 10 rt 90
> EOF
Keyword: fd
 Illegal: 3x00
0
Keyword: bk

setc 100
Keyword: int
 ID: xy3
 Keyword: fd
 10 rt 90
$

(OK – I cheated marginally: the first time around, I ran rmk lex.yy LDLIBS=-lfl, where rmk is a variant of make and the compilation rules in the directory use the command line shown. But I redid the compilations to get the error messages right, exactly as above.)

You might need to look at expanding your patterns to accept ‘one or more’ digits with [0-9]+ in place of just [0-9]. You might need to look at a rule dealing with unmatched characters. And personally, I go to great pains to avoid blanks immediately before newlines, so you would need to tighten up your print formatting to meet my criteria. However, that’s not germane to getting the program running.

Also, if you need to convert your file from DOS to Unix line endings, the easiest is the dos2unix command, if you have it on your machine. Otherwise, use:

$ tr -d '\015' < miniStarLogo.l > x
$ od -c x
0000000   %   {  \r  \n  \r  \n   /   *       C   o   m   m   e   n   t
...
0001560  \n   }  \r  \n
0001564
$ mv x miniStarLogo.l
$

I carefully added the carriage returns using vim and :set fileformat=dos; it would also be possible to undo it with vim and :set fileformat=unix. This is Unix so TMTOWTDI (There’s More Than One Way To Do It — the Perl motto), and I’m not even trying to use Perl.