Question

Below is a function for getting the next token in a program that does tokenizing. It currently is working but I’m still not sure if it’s correct for what my professor is asking for. If you look in the top comment section he says “Space for the return token should be dynamically allocated.”

Whenever I hear that I expect that I would have to use malloc but I’ve done it without using malloc. Am I correct to think that?

/*
 * TKGetNextToken returns the next token from the token stream as a
 * character string.  Space for the returned token should be dynamically
 * allocated.  The caller is responsible for freeing the space once it is
 * no longer needed.
 *
 * If the function succeeds, it returns a C string (delimited by '\0')
 * containing the token.  Else it returns 0.
 *
 * You need to fill in this function as part of your implementation.
 */
char *TKGetNextToken(TokenizerT *tk) {
char *sPtr, *tPtr, *delim, *temp = NULL, *ret = NULL;

sPtr = tk->sepr; //pointer to the separators

tPtr = tk->ts;
temp = tPtr;

while (tPtr[0] != '\0') //Scan tokenstream
{
    delim = tk->sepr;
    while (delim[0] != '\0') //scan separator stream 
    {
        if (*tPtr == *delim) //Matched with a separator
        {
            if (tPtr == temp) //Check if beginning of the tokenstream
            {
                //then skip over this character.
                temp++;
                break; //Break loop because it may skip checking a char with a prev delim.
            }
            else
            {
                //Cut off current position with null character and pass over it.
                *tPtr = '\0';
                tPtr++;

                if((ret = malloc(sizeof temp) + 1 * sizeof(char)) != NULL) //add 1 for null character '\0'
                    strcpy(ret, temp);

                tk->ts = tPtr; //In position for next token.

                return ret;
            }
        }
        else
        {
            delim++; //Go to next separator.
        }

    }

    tPtr++; //Go to next character.
    count++;
}

if((ret = malloc(sizeof temp) + 1 * sizeof(char)) != NULL) //add 1 for null character '\0'
    strcpy(ret, temp);

tk->ts = tPtr;
return ret;
}

EDIT: So I think I fixed it but curious, using malloc the way I did, doesn’t that only allocate space for the pointer address and not the entire token?

Answer 1

That is a possible implementation, but not the one your professor is asking for. Your professor wants you to allocate memory and copy the token into it, then return a pointer to that allocated memory.