Question

I have an input stream containing integers and special meaning characters ‘#’. It looks as follows:
... 12 18 16 # 22 24 26 15 # 17 # 32 35 33 ...
The tokens are separated by space. There’s no pattern for the position of ‘#’.

I was trying to tokenize the input stream like this:

int value;
std::ifstream input("data");
if (input.good()) {
  string line;
  while(getline(data, line) != EOF) {
    if (!line.empty()) {
      sstream ss(line);
      while (ss >> value) {
        //process value ...

      }
    }
  }
}

The problem with this code is that the processing stops when the first ‘#’ is encountered.

The only solution I can think of is to extract each individual token into a string (not ‘#’) and use atoi() function to convert the string to an integer. However, it’s very inefficient as the majority tokens are integer. Calling atoi() on the tokens introduces big overhead.

Is there a way I can parse the individual token by its type? ie, for integers, parse it as integers while for ‘#’, skip it. Thanks!

Answer 1

One possibility would be to explicitly skip whitespace (ss >> std::ws), and then to use ss.peek() to find out if a # follows. If yes, use ss.get() to read it and continue, otherwise use ss >> value to read the value.

If the positions of # don’t matter, you could also remove all '#' from the line before initializing the stringstream with it.