Question

I’m wondering if there’s a way in boost::spirit::lex to write a token value back to the input stream (possibly after editing) and rescanning again. What I’m basically looking for is a functionality like that offered by unput() in Flex.

Thanks!

Answer 1

I ended up implementing my own unput() functionality as follows:

   struct unputImpl
   {
      template <typename Iter1T, typename Iter2T, typename StrT>
      struct result {
         typedef void type;
      };

      template <typename Iter1T, typename Iter2T, typename StrT>
      typename result<Iter1T, Iter2T, StrT>::type operator()(Iter1T& start, Iter2T& end, StrT str) const {
         start -= (str.length() - std::distance(start, end));
         std::copy(str.begin(), str.end(), start);
         end = start;
      }
   };

   phoenix::function<unputImpl> const unput = unputImpl();

This can then be used like:

   this->self += lex::token_def<lex::omit>("{SYMBOL}\\(")
        [
           unput(_start, _end, "(" + construct<string>(_start, _end - 1) + " "),
           _pass = lex::pass_flags::pass_ignore
        ];

If the unputted string length is bigger than the matched token length, it will override some of the previously parsed input. The thing you need to take care of is to make sure the input string has sufficient empty space at the very beginning to handle the case where unput() is called for the very first matched token.