I’ve noticed that CSS using relative positioning will be handled very differently if my web page contains the HTML5 doctype header <!DOCTYPE HTML>. For example:
<html>
<body>
<img src="test.png" />
<span style="position: relative; top: -10;">TEST</span>
</body>
</html>
will render the word TEST aligned 10 pixels higher than its default position like so:

But if I add <!DOCTYPE HTML> to the top of the document and make no other changes:
<!DOCTYPE HTML>
<html>
<body>
<img src="test.png" />
<span style="position: relative; top: -10;">TEST</span>
</body>
</html>
then the relative positioning appears to have no affect on the word TEST:

This behavior is consistent in the latest versions of IE, Chrome, and Firefox on Windows. I’ve noticed other quirky behavior when using absolute positioning with and without the HTML5 doctype header.
Is there a fundamental change on how relative and absolute positioning is applied in HTML5?
This has nothing to do with HTML5; this is a difference between quirks mode and standards mode, both of which have existed for a long time.
Quirks mode can be triggered by a number of factors, one of which is shown in your question by the lack of a doctype declaration altogether. In quirks mode, unitless lengths are permitted (and treated as pixel lengths), which is why your relative positioning works.
In standards mode, unitless lengths aren’t permitted (unless they’re zero), so your
topdeclaration becomes invalid CSS, and is ignored. This is the same whether you’re using the HTML5 doctype or an HTML 4 or XHTML doctype. In fact, the HTML5 doctype doesn’t have any special meaning or purpose — it was created only to make browsers render in standards mode.