GUI based debugging in Linux using KGDB

Abstract

Linux Kernel and Device Driver Development is a very interesting experience. Having complete access to the entire source code is a boon. The wealth of technical know-how and knowing the best practices followed by experts by reading and following the Linux Kernel Source code is immense. Fresh device driver developers and students who want to learn device driver development are better off starting their lessons with Linux for the very fact that they would get the complete picture of what’s happening under the hood with the access to complete source code and immense literature on the web and in print.

But in today’s pressing requirement on the speed of development and steepness required in the learning curve actually vouches for the need to understand and use the tools available to make the kernel / driver development more fun and fast.

Linux Kernel Debugging

The most important part of any development lies in effective debugging of the issues that arise during the development and testing phases of the project. The Linux kernel and device driver development can become tough for the following reasons

1. Lack of complete understanding of the modules working in tandem with the module that is being developed. For e.g. when an engineer has developed the low level access routines for a flash device and is testing the device driver, if he encounters some problem or hangs in the file system level, he may not have the complete source level control over that part of the code and may find it difficult to debug that particular bug.

2. When the value of certain variables and arrays need to be checked during the kernel execution (or) certain flag values need to be dynamically changed during the kernel execution.

3. When there are a lot of switch cases, if-else blocks and there is a random entry into some block of code that causes the problem.

There are many such reasons which contribute to delays in kernel and driver debugging. Linux kernel debugging is an art and needs good amount of practice and enough development hours to master the debugging. Now this debugging can be made easier or assisted with the help of development tools and Linux debuggers.

Kernel GDB

The debugging of Linux kernel was more restricted to the PRINTK debug messages that get printed on the console (possibly a serial console for an embedded device). While this is effective at a lot of times, if we consider some of the cases mentioned above it would become a pain to put printk messages. Sometimes the messages pour out a lot causing considerable delay in execution and thereby hiding the bugs and at other times we might have missed the exact prints that need to be enabled, which is a very common scenario as we are debugging something unknown. So in such cases the kernel has to be rebuilt and re-deployed. This involves a lot of time and effort

To get around these inconveniences, we should have a debugger that should have the following features

1. Source code level Breakpoints

2. Controlled kernel execution

3. Variable watch at run-time

4. Register watch at run-time.

The Linux KGDB is a kernel level GDB that works well with the above features. The KGDB can be used to debug the kernel and device driver at runtime without the need to re-build multiple times. To enable the KGDB, the corresponding patch has to be applied and debugging has to be enabled in the kernel.

The command line KGDB Problem

The command KGDB is inconvenient in many ways. Its non integration with the source code and complete command line usage during debugging is a big put off. Getting the call stack details, live variable values, browsing the source code, etc. would be very difficult with the command line GDB that has virtually stopped many developers from using the debugger in the first place.

A lot of training hours had to be invested to get some good hold on the command line GDB and even then it is not near perfection.

Kernel GDB integrated with a GUI IDE

Integrating the Kernel GDB with and IDE like ECLIPSE would be the ideal testing environment and tool to use. The debugging opportunities it present with the integrated source level debugging is immense and would be the right tool for beginners and people who really want to speed up their debugging.

Let’s see a few things that the ECLIPSE environment integrated with the Kernel GDB can do when debugging the kernel. I believe this would help understand the key positive features of the ECLIPSE – KGDB combo.

The Linux kernel can be built using the eclipse IDE. The Linux kernel can be imported as a Makefile project and can be built using the eclipse IDE. Once the kernel is built, it can be loaded on to the target for debugging and there is an option in eclipse to connect the GDB server running with eclipse to connect to the GDB Client on the target board. How this is done varies from one target to another.

e-con Systems offers a 2.6.25 Linux Kernel with ECLIPSE and Kernel GDB support for their Regulus PXA270 SOM based development board. The snapshots captured and presented here would be from that platform.

Controlling the kernel execution – Break Points

More often than not during my early driver development stages, I used to feel that the delay in addressing a bug was due to the lack of control over the execution flow. The time to taken to verify a path of execution practically was huge and instant debug path changes, as the execution path changed was next to impossible. Let me explain how this problem is addressed with the KGDB-ECLIPSE (IDE) combo.

The Linux kernel built with debugging support waits for the GDB server connection when booting up. The GDB server running on the development PC has to connect to the kernel that is booting over the serial debug port and has to proceed from there on based commands from the GDB server.

The GDB server runs as part of the eclipse IDE and controls the kernel running on the kernel.

The source level breakpoints can be enabled on the IDE with the ease of a mouse click. One can enable and disable breakpoints at runtime providing immense freedom to control the execution of the kernel.

Below is a snapshot of the IDE window where you can see the source code on the window

The marked portion on the left side of the IDE interface across the code indicates the breakpoints that are being set and enabled. The kernel flow would stop at these points precisely. The snapshot shows the kernel execution stopped at the regulus_init start

Once the execution stops at the breakpoint, a number of parameters can be analyzed as we would see in the following sections. The execution can be continued by clicking on the resume.

The debugger also provided options to

The snapshot below provides the options for the RUN.

Understanding the code flow – Call Stack

When the kernel execution is stopped on a breakpoint, the developer has an option to view the call stack and trace the flow thus far in the execution.

In the picture below, the top “DEBUG” portion of the IDE indicates the callstack. It shows that the regulus_init() function was called by customize_machine() which was called by Kernel_init. Clicking on the respective function would open the function definition source code in the source code window.

Peeping into the Memory, Variables and Registers – Watching

Another problem that we normally face is the memory contents. Many a time we feel the necessity to have a peek into some memory buffer or the other. For eg.., assume that we doubt an endian problem in the Ethernet buffer and want to have a look at the buffer contents at the receive path. We could place a break point at the Rx path of the driver and use the eclipse-GDB to view the memory contents.

The snapshot below shows the pointer and the memory contents of the buffer clearly in ASCII. It also shows in HEX. Please observe the marked areas in the snapshot to get a clear idea.

Similarly the variables can be viewed from the top section of the UI. It can be noted that the value of the variable “testpointer” was found using the watchwindow at the top right section.

This would display the live value of the variables in the kernel. This is of great help when it comes to FLAGS and kernel decision paths (if, elses… and Switch cases…). The value of the variable gets updated after we break at a particular section of the code.

Similarly the core registers can also be viewed. I am using a ARM core processor and so the ARM core registers can be viewed.

Changing the variables at Run time

The need to change a variable at runtime is another feature that we may like to use. This variable may be a debug switch that can be turned ON and OFF at runtime or this could be a value of the number of loops a particular test should run. If this is possible then the rebuilding of the kernel could be avoided and debugging could be very fast.

This is possible with this debugging setup.

In this snapshot we could clearly see the “testvariable” value being changed from 1 to 100 just by clicking on the variable.

Conclusion

With this I would like to end of the Part-1 of the Debugging with Eclipse-GDB. I have used the e-con Systems’ eSOM270 module based Regulus Development board along with the BSP and debugging environment provided by them. Based on the responses to this article I am planning to write more detailed articles on how I Debug Ethernet drivers, File System drivers, etc on the regulus platform.