Question

I would like to write simple programs (console input/output) for Windows using x86 assembly, mainly because I am just curious. It would be great if anybody could point me in the right direction. I already have a fairly good understanding of some of the more simple x86 instructions, the functions of the registers, etc. but it remains a mystery to me how programs interface with the operating system and use standard input and output. I know that these things have to do with libraries such as advapi32.dll and kernel32.dll, and that there are associated static library .lib files which enable compilers to use these dynamically linked libraries, but other than that I have little idea how this happens. I’m even fuzzy on how header files in languages like C use .lib files.

Answer 1

Perhaps it’s easiest to give directions for building some simple programs, and let you extrapolate from there. First you need some source code:

.386
.MODEL flat, stdcall

; This is what would come from a header -- a declaration of a the Windows function:
MessageBoxA PROTO near32 stdcall, window:dword, text:near32,
        windowtitle:near32, style:dword

.stack 8192

.data
message db "Hello World!", 0
windowtitle   db "Win32 Hello World.", 0

.code
main proc
        invoke MessageBoxA, 0, near32 ptr message, near32 ptr windowtitle, 0
        ret
main endp
        end main

To build that, we’d invoke masm like:

ml hello32.asm -link -subsystem:windows user32.lib

That’s telling it the file to assemble, and when it links, telling it to link it as being for the Windows subsystem (the primary alternative would be -subsystem:console) and to link with user32.lib. The latter provides the definition of MessageBoxA for us.

A similar program that writes to the console is (oddly enough) a tiny bit more complex:

.386
.MODEL flat, stdcall

getstdout = -11

WriteFile PROTO NEAR32 stdcall, \
        handle:dword,           \
        buffer:ptr byte,        \
        bytes:dword,            \
        written: ptr dword,     \
        overlapped: ptr byte

GetStdHandle PROTO NEAR32, device:dword

ExitProcess PROTO NEAR32, exitcode:dword

.data
message db "Hello World!", 13, 10
msg_size dd $ - offset message

.data?
written  dd ?

.code
main proc   
    invoke GetStdHandle, getstdout

    invoke WriteFile,                   \
           eax,                         \
           offset message,              \
           msg_size,                    \
           offset written,              \
           0

    invoke ExitProcess, 0
main endp
        end main

Building is pretty much the same, except that this uses the console, so we specify the console subsystem, and the functions we’re using are defined in the kernel:

ml hello_console.asm -link -subsystem:console kernel32.lib

A header would include the equivalent of the declarations I’ve given above for MessageBoxA, GetStdHandle, WriteFile, etc. Each header would normally have a lot more of them though — for example, all the functions in kernel32 might be in a single header.

As far as libraries go, the mechanism involved is somewhat involved, but mostly irrelevant. To get the job done, you look at (for example) MSDN, and see what library says to link, and add that on your command line.

The more complex explanation is that at least when you link against a static library, it simply finds any functions you’ve called, and puts a copy of each into your executable/DLL. If (like above) you’re using code that’s in a DLL, it basically just puts a record into the executable that tells what function in what DLL it depends upon. Then, when you load/run the program, the loader looks for all the DLLs your program depends upon, and loads them (recursively loading anything they depend upon as well, of course). Then the loader fixes up those references, so the reference to the function in the DLL gets filled in with whatever address has been assigned to that function in that DLL.