Question

I’m writing my own protected mode operating system and need to know how to modify the interrupt table so that certain ones (print string, etc.) are redirected to the command prompt application or other applications. How do I modify the table?

Answer 1

First of all, using interrupts for system calls is really not the “nice” way to actually implement them anymore. If you’re targeting x64, then you should definitely look into SYSCALL and SYSRET, which enable very convenient (from the user’s point of view) calling of kernel code, provided that you’re willing to deal with some setup routines needed to use those two. If that isn’t an option, consider mapping some of the kernel code into processes’ virtual memory space. Google for linux-gate.so.1 to read about an example implementation of that.

But, if you still want to implement syscalls via interrupts, then read on. Of course, you’ll probably need to modify the interrupt table either way in order to actually interact with the hardware.

In the Protected Mode of x86, interrupts are placed in the IDT, which is short for an Interrupt Descriptor Table. That table can contain either a Task Gate Descriptor, an Interrupt Gate Descriptor, or a Trap Gate Descriptor. The table is indexed in a similar way to the Real Mode Interrupt Vector Table – e.g. the n-th entry of the table contains the descriptor that will be used to service the interrupt n. Interrupt and trap gates are really just pointers to the interrupt routine code with some extra data, while a task gate contains a Task State Segment Selector, which is a selector of the task that is supposed to handle that interrupt. In 64-bit mode, only interrupt/trap gates are supported due to the deprecation of hardware task handling.

I don’t really see the point in simply rephrasing the Intel manual, which explains these issues very clearly, so I suggest you just have a read (though you should have those manuals by now). Interrupt handling is described in chapter 6 of that document.