I noticed using JavaScript I can produce HTML with mismatching tags that still pass the w3 validator. For example
<script type="text/javascript>
document.write("<h4>This is large text<h2></h4> This is small text")
</script>
has unnested </h4> and no closing <h2> but the document still passes as HTML 5 (or whatever). Can this be an issue regarding browser compatibility and why is it not detected?
The w3 validator doesn’t execute Javascript, nor should it. It only checks code that you have explicitly supplied to it.