Question

I’m writing a simple shell program in C. As part of the program, I’m writing a tokenizer to break up user input into tokens. Each time the tokenizer is called, it returns the next token. The code for the tokenizer looks like this:

char* nextToken(char string[])
{
char token[50]= {}; //stores the current token
//More code here, will post if necessary
puts(token);
return token;
}

int main()
{
char inputString[] = "cpgm one two <three| script a b >file";
char *token = nextToken(inputString);
while(*token!='\0') //I'm using a char[] with a single null character as a delimiter to indicate the last token has been reached
{
    token = nextToken(inputString);
}

}

When I run the program like this, the “puts” works properly; that is to say, each time the function is called, the appropriate token is printed in the correct order, like this:

cpgm
one
two
<
three
|
script
a
b
>
file

However, when I remove the “puts” line in nextToken and put it in the main function, like this:

 while(*token!='\0')
 {
     puts(token);
     token = nextToken(inputString);
 }

and try to print the token from my main function instead, all I see is a list of weird characters, like this:

�bHd?
�bH
�bH
�
�bHd
�
�bHd?

�
�
�
�
�bHd?

Any idea why this is happening?

Thanks.

Answer 1

You are allocating your string token on the stack. This means that when you return it, it will be deallocated, and can’t be used any more. You should allocate memory for your string instead:

char *token = malloc(50);

You also then have to remember to free the string later, once you’re done with it, by calling

free(token);