Question

I’d like to try writing my own minimal NT subsystem on Windows 7 for purely educational purposes — something like a bare-bones equivalent of the posix.exe in Microsoft’s Subsystem for Unix-based Applications.

But I can’t seem to find any public documentation on this topic. What API does a subsystem need to implement? How does it get registered with Windows? How does the subsystem image need to be built (what flags need to be set in the PE header, etc.)?

I’d most like to find a book or web site with an overview of the entire subject, or even the source code for a “hello world” NT subsystem that someone else has written. But anything at all would be appreciated if you can point me in the right direction here…

Answer 1

Here are the major components of a subsystem:

• User-mode server. The server creates a (A)LPC port and listens for and handles client requests.
• User-mode client DLL. In the DLL_INIT_ROUTINE, you can connect to the port set up by the server. This DLL will expose your subsystem’s API, and some functions will require communication with the server.
• Kernel-mode support driver (you might not need this).

You will want to store process or thread state in either your server or driver. If you’re storing it in the server, you might need something like NtRegisterThreadTerminatePort to ensure you get to clean up when a process or thread exits. If you’re using a driver, you need PsSetCreateProcessNotifyRoutine.

And lastly, if you’re on XP and below, you can add new system calls. You can do this by calling KeAddSystemServiceTable. To invoke the system calls from user-mode, you need to create stubs like this (for x86):

; XyzCreateFooBar(__out PHANDLE FooBarHandle, __in ACCESS_MASK DesiredAccess, ...)
mov     eax, SYSTEM_CALL_NUMBER
mov     edx, 0x7ffe0300
call    [edx]
retn    4

On Vista and above you can no longer add new system service tables because there is only room for two: the kernel’s system calls and win32k’s system calls.

After a bit of Googling I found this: http://winntposix.sourceforge.net/. I think it’s very similar to what you’re looking for, and uses a lot of the things I have mentioned.