It seems like every question on stackoverflow where the asker is using regex to grab some information from HTML will inevitably have an ‘answer’ that says not to use regex to parse HTML.
Why not? I’m aware that there are quote-unquote ‘real’ HTML parsers out there like Beautiful Soup, and I’m sure they’re powerful and useful, but if you’re just doing something simple, quick, or dirty, then why bother using something so complicated when a few regex statements will work just fine?
Moreover, is there just something fundamental that I don’t understand about regex that makes them a bad choice for parsing in general?
Entire HTML parsing is not possible with regular expressions, since it depends on matching the opening and the closing tag which is not possible with regexps.
Regular expressions can only match regular languages but HTML is a context-free language and not a regular language (As @StefanPochmann pointed out, regular languages are also context-free, so context-free doesn’t necessarily mean not regular). The only thing you can do with regexps on HTML is heuristics but that will not work on every condition. It should be possible to present a HTML file that will be matched wrongly by any regular expression.