Question

I would like to do some parsing and tokenizing in c++ for learning purposes. Now I often times came across bison/yacc and lex when reading about this subject online.
Would there be any mayor benefit of using those over for instance a tokenizer/parser written using STL or boost::regex or maybe even just C?

Answer 1

I recently undertook writing a simple lexer and parser.

It turned out that the lexer was simpler to code by hand. But the parser was a little more difficult. My Bison-generated parser worked almost right off the bat, and it gave me a lot of helpful messages about where I had forgotten about states. I later wrote the same parser by hand but it took a lot more debugging before I had it working perfectly.

The appeal of generating tools for lexers and parsers is that you can write the specification in a clean, easy-to-read language that comes close to being a shortest-possible rendition of your spec. A hand-written parser is usually at least twice as big. Also, the automated parser (/lexer) comes with a lot of diagnostic code and logic to help you get the thing debugged.

A parser/lexer spec in BNF-like language is also a lot easier to change, should your language or requirements change. If you’re dealing with a hand-written parser/lexer, you may need to dig deeply into your code and make significant changes.

Finally, because they’re often implemented as finite state machines without backtracking (gazillions of options on Bison, so this is not always a given), it’s quite possible that your auto-generated code will be more efficient than your hand-coded product.