There is no day on SO that passes without a question about parsing (X)HTML or XML with regular expressions being asked.
While it’s relatively easy to come up with examples that demonstrates the non-viability of regexes for this task or with a collection of expressions to represent the concept, I could still not find on SO a formal explanation of why this is not possible done in layman’s terms.
The only formal explanations I could find so far on this site are probably extremely accurate, but also quite cryptic to the self-taught programmer:
the flaw here is that HTML is a Chomsky Type 2 grammar (context free
grammar) and RegEx is a Chomsky Type 3 grammar (regular expression)
or:
Regular expressions can only match regular languages but HTML is a
context-free language.
or:
A finite automaton (which is the data structure underlying a regular
expression) does not have memory apart from the state it’s in, and if
you have arbitrarily deep nesting, you need an arbitrarily large
automaton, which collides with the notion of a finite automaton.
or:
The Pumping lemma for regular languages is the reason why you can’t do
that.
[The majority of the above explanation link to wikipedia pages, but these are not much easier to understand than the answers themselves].
What is a translation in layman’s terms of the formal explanations given above of why it is not possible to use regex for parsing (X)HTML/XML?
I am looking for a translation that also briefly explains the concepts it tries to translate: at the end of an answer, the reader should have a rough idea –for example–of what "regular language" and "context-free grammar" mean.
Concentrate on this one:
The definition of regular expressions is equivalent to the fact that a test of whether a string matches the pattern can be performed by a finite automaton (one different automaton for each pattern). A finite automaton has no memory – no stack, no heap, no infinite tape to scribble on. All it has is a finite number of internal states, each of which can read a unit of input from the string being tested, and use that to decide which state to move to next. As special cases, it has two termination states: “yes, that matched”, and “no, that didn’t match”.
HTML, on the other hand, has structures that can nest arbitrarily deep. To determine whether a file is valid HTML or not, you need to check that all the closing tags match a previous opening tag. To understand it, you need to know which element is being closed. Without any means to “remember” what opening tags you’ve seen, no chance.
Note however that most “regex” libraries actually permit more than just the strict definition of regular expressions. If they can match back-references, then they’ve gone beyond a regular language. So the reason why you shouldn’t use a regex library on HTML is a little more complex than the simple fact that HTML is not regular.