Question

For my homework project, I had to read in a book and then parse it word for word, listing occurrences of each word and the respective line it was found at. I’m done with the project now and looking to optimize my code by reading other people’s use of the STL (we are supposed to rely heavily on the STL for this assignment).

This is not regarding the homework itself, but two attempts I made to read in each word that resulted in the stream eating my newlines. Without them, I couldn’t keep track of which line I’m on in the document I’m reading, therefore eliminating the use of such solutions.

Attempt One:

string word;
stringstream ss;
ss << document; // Document was read from file and stored in a "string document"
while(ss)
{
    ss >> word;
    // Work with the word.
    // Oops! No newlines anymore! :(
}

Attempt Two:

ifstream ifs(filename.c_str());
if(ifs.is_open)
{
    typedef istream_iterator<string> string_input;
    vector<string> words;
    copy(string_input(ifs), strin_input(), back_inserter(words));

    // Oops! The newlines are gone here too! :(
}

My current solution isn’t as pretty as I want it to be, and I want some more STL magic in it (just for the sake of learning some neat STL tricks and getting even more comfortable with it)

Current solution:

Read the file:

std::ostringstream os;
ss << ifstream(filename.c_str()).rdbuf();
return ss.str();

And split by line and word:

size_t oldNewLine = 0;
size_t newLine = document_.find('\n', oldNewLine);
while(newLine != string::npos)
{
    string documentLine = document_.substr(oldNewLine, (newLine - oldNewLine));
    vector<string> words = Utility::split(documentLine);

   // Work with each individual word
   // Yay! I can keep track of the line number now! :)

   oldNewLine = newLine + 1; // Curse you, off by one error!
   newLine = document_.find('\n', oldNewLine);
}

The reading of the file is short, succinct and very readable imo, but the splitting of each line and word is tedious. I wish I could just immediately read each word from file, preserving the newline. There has to be a short, sweet way to do this!

So, how to do it?

Answer 1

You can parse the words line by line:

std::string line;
vector<string> words;

while(std::getline(ifs, line))
{
    std::stringstream linestream(line);

    copy(std::istream_iterator<std::string>(linestream),
         std::istream_iterator<std::string>(),
         std::back_inserter(words));

    ++lineCount;
}