Question

I have some text (meaningful text or arithmetical expression) and I want to split it into words.
If I had a single delimiter, I’d use:

std::stringstream stringStream(inputString);
std::string word;
while(std::getline(stringStream, word, delimiter)) 
{
    wordVector.push_back(word);
}

How can I break the string into tokens with several delimiters?

Answer 1

Assuming one of the delimiters is newline, the following reads the line and further splits it by the delimiters. For this example I’ve chosen the delimiters space, apostrophe, and semi-colon.

std::stringstream stringStream(inputString);
std::string line;
while(std::getline(stringStream, line)) 
{
    std::size_t prev = 0, pos;
    while ((pos = line.find_first_of(" ';", prev)) != std::string::npos)
    {
        if (pos > prev)
            wordVector.push_back(line.substr(prev, pos-prev));
        prev = pos+1;
    }
    if (prev < line.length())
        wordVector.push_back(line.substr(prev, std::string::npos));
}