Question

What i think is that there is a bug in the boost token ids. But I’m not sure.
The boost token iterators are not able to detect the T_CONTLINE token which is ‘\ \’ followed by ‘\n’ See: http://www.boost.org/doc/libs/1_46_1/libs/wave/doc/token_ids.html

For the sample test, I have a test file: testfile.h

  1 #define Funtion(x) X + \
  2                       Y
  3 
  4 #define MYVAR 100+\
  5               200

And here is the program which looks for the T_CONTLINE

  1 #include <iostream>
  2 #include <fstream>
  3 
  4 #include <boost/wave/cpplexer/cpp_lex_token.hpp>
  5 #include <boost/wave/cpplexer/cpp_lex_iterator.hpp>
  6 
  7 typedef boost::wave::cpplexer::lex_token<> token_type;
  8 typedef boost::wave::cpplexer::lex_iterator<token_type> token_iterator;
  9 typedef token_type::position_type position_type;
 10 
 11 int main()
 12 {
 13   const char* infile = "testfile.h";
 14   std::string instr;
 15   std::ifstream gmstream(infile);
 16   if(!gmstream.is_open()) {
 17       std::cerr << "Could not open file: "<< infile<<"\n";
 18   }
 19   gmstream.unsetf(std::ios::skipws);
 20   instr = std::string(std::istreambuf_iterator<char>(gmstream.rdbuf()),
 21       std::istreambuf_iterator<char>());
 22 
 23   position_type pos(infile);
 24   token_iterator  it = token_iterator(instr.begin(), instr.end(), pos,
 25       boost::wave::language_support(
 26         boost::wave::support_cpp|boost::wave::support_option_long_long));
 27   token_iterator end = token_iterator();
 28 
 29   boost::wave::token_id id = *it;
 30 
 31   while(it!=end) {
 32     if(id == boost::wave::T_CONTLINE) {
 33       std::cout<<"Found Contline";
 34     }
 35     ++it;
 36     id = *it;
 37   }
 38 return 0;
 39 }

But I don’t get any output.

I’m using boost_1_47_1, and gcc-4.5

EDIT:
Actually there was a similar bug-report posted here:

https://svn.boost.org/trac/boost/ticket/5569

See the changelog: http://www.boost.org/doc/libs/1_47_0/libs/wave/ChangeLog

This bug is reported to be solved with the latest boost 1.47.0 So i installed the latest boost 1.47.0 but still the problem remains.

Answer 1

Yes, this token is not exposed on the iterator level, it is just handled internally very early in the processing. It’s not a bug, but expected behavior. I will add a note to the documentation.