Question

I am writing essentially the equivalent of memset() using ASM. The code I have written works, except that when I try to restore the stack, it crashes with an access violation. I have put the code into MSVC using inline assembly, so I can debug it.

The problem occurs when the function returns. However, when I take the add esp, 4 line out, the code executes correctly, but after the main() function has returned, MSVC says that the stack around a variable was corrupted.

I am reluctant to continue without the add esp, 4 as I know this will cause problems later.

How would I go about fixing this?

int main(int argc, char **argv)
{
    char szText[3];


    /*__asm
    {
        push 3
        mov edx, 65
        lea ecx, szText
        call memset
    }*/

    memset((void*)&szText, 'A', 3);

    return 42;
}

void __declspec(naked) __fastcall memset(void *pDest, int iValue, int iSize)
{
    __asm
    {
        ; Assume the pointer to the memory is stored in ECX
        ; Assume the value is stored in EDX
        ; Assume the size of the block is stored on the stack

            mov eax, esi        ; Put ESI somewhere it won't be touched (I think)

            mov esi, ecx        ; Move the address of the memory into ESI
            xor ecx, ecx        ; Zero ECX

            mov ecx, [esp+4]    ; Get the size of the block into ECX. ECX is our loop counter

        memset_count:
            cmp ecx, 0          ; If we are at the end of the block,
            jz memset_return    ; Jump to return

            mov [esi], edx      ; Move our value into the memory
            inc esi             ; Otherwise, increment out position in the memory
            dec ecx             ; Decrement out counter
            jmp memset_count    ; Start again

        memset_return:
            mov esi, eax        ; Restore ESI
            add esp, 4          ; Clean up the stack
            ret

    }
}

Answer 1

        memset_return:
            mov esi, eax        ; Restore ESI
            add esp, 4          ; Clean up the stack
            ret

That’s wrong, you didn’t subtract 4 from ESP in the function body. You’ll actually skip the return address and RET pops the argument from the stack and jumps to its value. Kaboom. Fix:

        memset_return:
            mov esi, eax        ; Restore ESI
            ret 4

You will also have to write a function prototype so that the compiler knows that the calling convention for the function is non-standard. It will generate the wrong code at the call site if it doesn’t know. Paste this before the main method:

 void  __fastcall memset(void *pDest, int iValue, int iSize);

And avoid naming it “memset”, that’s an intrinsic function that’s also emitted by the code generator. It will call your function instead of the standard one. Which will go badly, the standard one has a very different signature. Pick another name to avoid this almost impossible to debug mishap.