Question

I’ve been playing with assembly for a while and looking at some code.
in which AL is first set to 0x84 then cmp AL, 0x30 is used.
This instruction then triggers the Overflow flag.

From what I read CMP is supposed to subtract the second number from the first then set the flags, in that case it should be 0x84-0x30 the result is 0x54 and there is no overflow.

Answer 1

There’s only no overflow if you’re interpret those values as unsigned numbers – if you interpret your 0x84 as signed, there’s definitely overflow:

1. 0x84 interpreted as a signed 8-bit value is -124
2. 0x30 interpreted as a signed 8-bit value is 48
3. -124 – 48 = -172

-172 is outside of the range of a signed 8-bit value (-128 to +127) and that’s why the OF flag gets set. You should check CF which indicates unsigned overflow.

From the Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 2 for CMP:

The comparison is performed by subtracting the second operand from the first operand and then setting the status flags in the same manner as the SUB instruction.

and for SUB:

The SUB instruction performs integer subtraction. It evaluates the result for both signed and unsigned integer operands and sets the OF and CF flags to indicate an overflow in the signed or unsigned result, respectively. The SF flag indicates the sign of the signed result.