Question

I have a device driver that is freezing the OS. The mouse wont even move. I am trying to debug this issue and I believe one good approach is to use gdb with qemu, two things I have never used before. Is there a better approach?

So first I need to compile the kernel with debug symbols which I have done already.

Now, there is a new file that is generated called vmlinux that is located in that same folder as the source. It seems that I also need a bzImage file according to this so I can run the newly compiled kernel using:

qemu-system-i386 -kernel bzImage 

or in debug mode

qemu-system-i386 -s -S -kernel bzImage

I cannot locate the bzImage file. Where do I find it or what is missing here? Is the bzImage referring to the OS Image I created using qemu-img create?

Also, what I do not understand is that now the kernel is compiled (vmlinux) how does I run it with qemu? So my question is when I run it with qemu or the debugger is the kernel running as an app in my main OS?

also how can I install my device driver? My understanding the kernel is not Ubuntu so there is no UI?

Also, I installed qemu and when I type qemu I get command not found. I am guessing I have to pick a specific processor emulator as in qemu-system-i386, qemu-system-x86_64, or qemu-x86_64?

How is qemu different or similar to the kvm command?

Thanks.

Answer 1

So, if I understand the problem correctly, you have a kernel module that needs no specific hardware. When you are working with the module, the system freezes but the kernel log contains nothing special.

The following may be helpful.

Getting the log

The symptoms you described may still be a result of a kernel oops or panic. The logging facilities sometimes die before they can output the information about the error to the log file. You may try to output the log via a serial port, this should be more reliable.

As your kernel module does not need any specific hardware, the easiest way is probably to install the same Linux distro as you use to a virtual machine and connect the virtual serial port (COM) of that machine to a pipe on your host system.

This is usually quite easy to do. For example, this blog post contains the detailed instructions in case the host OS and the guest OS are Ubuntu 11.10.

VirtualBox is used there to manage the virtual machines. If you prefer QEMU, this should be possible as well. I suppose it is a bit easier to go with VirtualBox though but it is a matter of personal preference.

Basically, you need to perform the following steps.

• Create a virtual machine and install the Linux distro you need as a guest OS there.
• Enable a serial port (COM1, …) in the configuration of the virtual machine and configure it to connect to a special file on the host ("host pipe"), say /tmp/vbox_serial.
• Start the guest OS and adjust its boot options: at least, add console=ttyS0,115200 or something like that to the kernel options in the boot loader menu.
• On the host, start minicom, socat or whatever else to read from /tmp/vbox_serial.
• That is it. Now you should get the kernel log of the guest OS pouring to your host system via /tmp/vbox_serial. If the guest system crashes then, you will get the log even if it is not saved into a file on the guest itself.

To make things easier, you may use socat on your host system rather than minicom that the author of that blog post suggests. The power of minicom is probably not needed here.

This way, you can use socat and tee to save the log to guest.log file while still outputting it to the console:

socat /tmp/vbox_serial - | tee guest.log

If there was a kernel oops or panic, the backtrace in the log usually helps to find out what
has gone wrong.

Detecting Deadlocks

If you have obtained the full log via a serial connection or some other means and still there is nothing suspicious there and you suspect there has been a deadlock in the kernel,
lockdep tool may help. It is included into the kernel (but you may need to rebuild the kernel with CONFIG_LOCKDEP_SUPPORT=y).

Lockdep detects the potential deadlocks and outputs the results to the kernel log. This presentation may help you analyse its output.

Tracing Facilities

If you need tracing of some events in the kernel to debug your system, there are some tools that could be handy.

• Kprobes – a kind of breakpoints you can set in almost arbitrary place in the kernel. Can be used to trace function calls among other things, with a moderate performance impact.
• SystemTap – a powerful system to analyze what is going on in the kernel. Part of it is based on Kprobes.
• Ftrace – a tracing system included into the kernel, incurs less overhead than Kprobes if that matters.