Question

I am trying to tokenize a string but I need to know exactly when no data is seen between two tokens. e.g when tokenizing the following string “a,b,c,,,d,e” I need to know about the two empty slots between ‘d‘ and ‘e‘… which I am unable to find out simply using strtok(). My attempt is shown below:

char arr_fields[num_of_fields];
char delim[]=",\n";
char *tok;
tok=strtok(line,delim);//line contains the data

for(i=0;i<num_of_fields;i++,tok=strtok(NULL,delim))
{
    if(tok)
        sprintf(arr_fields[i], "%s", tok);
    else
        sprintf(arr_fields[i], "%s", "-");          
}

Executing the above code with the aforementioned examples put characters a,b,c,d,e into first five elements of arr_fields which is not desirable. I need the position of each character to go in specific indexes of array: i.e if there is a character missing between two characters, it should be recorded as is.

Answer 1

7.21.5.8 the strtok function

The standard says the following regarding strtok:

[#3] The first call in the sequence searches the string
pointed to by s1 for the first character that is not
contained in the current separator string pointed to by s2.
If no such character is found, then there are no tokens in
the string pointed to by s1 and the strtok function returns
a null pointer. If such a character is found, it is the
start of the first token.

In the above quote we can read you cannot use strtok as a solution to your specific problem, since it will treat any sequential characters found in delims as a single token.

---

Am I doomed to weep in silence, or can somebody help me out?

You can easily implement your own version of strtok that does what you want, see the snippets at the end of this post.

strtok_single makes use of strpbrk (char const* src, const char* delims) which will return a pointer to the first occurrence of any character in delims that is found in the null-terminated string src.

If no matching character is found the function will return NULL.

---

strtok_single

char *
strtok_single (char * str, char const * delims)
{
  static char  * src = NULL;
  char  *  p,  * ret = 0;

  if (str != NULL)
    src = str;

  if (src == NULL)
    return NULL;

  if ((p = strpbrk (src, delims)) != NULL) {
    *p  = 0;
    ret = src;
    src = ++p;

  } else if (*src) {
    ret = src;
    src = NULL;
  }

  return ret;
}

sample use

  char delims[] = ",";
  char data  [] = "foo,bar,,baz,biz";

  char * p    = strtok_single (data, delims);

  while (p) {
    printf ("%s\n", *p ? p : "<empty>");

    p = strtok_single (NULL, delims);
  }

output

foo
bar
<empty>
baz
biz