Question

In my C++ code, I want to read from a text file (*.txt) and tokenize every entry. More specifically, I want to be able to read individual words from a file, such as “format”, “stack”, “Jason”, “europe”, etc.

I chose to use fstream to perform this task, and I do not know how to set it’s delimiter to the ones I want to use (space, \n, as well as hyphens and even apostrophes as in “Mcdonal’s”). I figured space and \n are the default delimiters, but hyphens are not, but I want to treat them as delimiters so that when parsing the file, I will get words in “blah blah xxx animal–cat” as simply “blah”, “blah”, “xxx”, “animal”, “cat”.

That is, I want to be able to get two strings from “stack-overflow”, “you’re”, etc, and still be able to maintain \n and space as delimiters at the same time.

Answer 1

An istream treats “white space” as delimiters. It uses a locale to tell it what characters are white space. A locale, in turn, includes a ctype facet that classifies character types. Such a facet could look something like this:

#include <locale>
#include <iostream>
#include <algorithm>
#include <iterator>
#include <vector>
#include <sstream>

class my_ctype : public
std::ctype<char>
{
    mask my_table[table_size];
public:
    my_ctype(size_t refs = 0)  
        : std::ctype<char>(&my_table[0], false, refs)
    {
        std::copy_n(classic_table(), table_size, my_table);
        my_table['-'] = (mask)space;
        my_table['\''] = (mask)space;
    }
};

And a little test program to show it works:

int main() {
    std::istringstream input("This is some input from McDonald's and Burger-King.");
    std::locale x(std::locale::classic(), new my_ctype);
    input.imbue(x);

    std::copy(std::istream_iterator<std::string>(input),
        std::istream_iterator<std::string>(),
        std::ostream_iterator<std::string>(std::cout, "\n"));

    return 0;
}

Result:

This
is
some
input
from
McDonald
s
and
Burger
King.

istream_iterator<string> uses >> to read the individual strings from the stream, so if you use them directly, you should get the same results. The parts you need to include are creating the locale and using imbue to make the stream use that locale.