Question

The following C++ program

#include <iostream>
#include <boost/tokenizer.hpp>
using namespace std;
int main()
{
    typedef boost::tokenizer<boost::char_separator<char> > tokenizer;

    string s ("(0,30,0)");

    tokenizer tokens(s.substr(1,s.size()-2), boost::char_separator<char>(","));
    for (tokenizer::iterator tok_iter = tokens.begin(); tok_iter != tokens.end(); ++tok_iter)
        cout<<*tok_iter<<endl;

    return 0;
}

has a problem since it outputs 0 0 0 instead of 0 30 0…

But if I slightly change it like that:

#include <iostream>
#include <boost/tokenizer.hpp>
using namespace std;
int main()
{
    typedef boost::tokenizer<boost::char_separator<char> > tokenizer;

    string s ("(0,30,0)");

    //Modified lines:
    string t (s.substr(1,s.size()-2));
    tokenizer tokens(t, boost::char_separator<char>(","));

    for (tokenizer::iterator tok_iter = tokens.begin(); tok_iter != tokens.end(); ++tok_iter)
        cout<<*tok_iter<<endl;

    return 0;
}

It works correctly! How is this possible ? At first sight, I would think that the two versions are the same…

Answer 1

In your first example, s.substr(1,s.size()-2) returns a temporary object and its lifetime expires at the end of the expression. However, boost::tokenizer has a constructor that takes a const reference, which means that the lifetime of the object will now be extended for as long as that reference exists. The problem is that boost::tokenizer only stores iterators to the passed in reference, so the reference to the object expires and the iterators are now invalid.