Kernel debugging with Kprobes
Insert printk's into the Linux kernel on the fly
(), Developer, Linux Technology
Center, IBM India Software Labs
19 Aug 2004
Collecting debugging information from the Linux™ kernel using printk is a well-known method -- and with Kprobes, it can be done without the need to constantly reboot and rebuild the kernel. Kprobes, in combination with 2.6 kernels, provides a lightweight, non-disruptive, and powerful mechanism to insert printk's dynamically. Logging debug info, such as the kernel stack trace, kernel data structures, and registers, has never been so easy!
Kprobes is a simple and lightweight mechanism in Linux that allows you to insert breakpoints into a running kernel. Kprobes provides an interface to break into any kernel routine and collect information non-disruptively from the interrupt handler. Debugging information, such as processor registers and global data structures, can be easily collected using Kprobes. Developers can even use Kprobes to modify register values and global data structure values.
To accomplish this, Kprobes inserts a probe by dynamically writing breakpoint instructions at a given address in the running kernel. Execution of the probed instruction results in a breakpoint fault. Kprobes hooks in to the breakpoint handler and collects the debugging information. Kprobes can even single-step probed instructions.
To install Kprobes, download the latest patch from the Kprobes home page (see for a link). The tarred file will be named something along the lines of kprobes-2.6.8-rc1.tar.gz. Untar the patch and apply it to the Linux kernel:
$tar -xvzf kprobes-2.6.8-rc1.tar.gz
$cd /usr/src/linux-2.6.8-rc1
$patch -p1 < ../kprobes-2.6.8-rc1-base.patch
Kprobes makes use of the SysRq key, an artifact from the days of DOS that has found many new uses under Linux (see ). You'll find the SysRq key to the left of the Scroll Lock key; it's often also labeled Print Screen. To enable the SysRq key for Kprobes, apply the kprobes-2.6.8-rc1-sysrq.patch patch:
$patch -p1 < ../kprobes-2.6.8-rc1-sysrq.patch
Configure the kernel with make xconfig/ make menuconfig/ make
oldconfig
and enable CONFIG_KPROBES
and
CONFIG_MAGIC_SYSRQ
flags. Build and boot into the new kernel. You
are now ready to insert printk's and collect debugging information dynamically
and unobtrusively by writing simple Kprobes modules.
|
For each probe, you will need to allocate the structure struct kprobe
kp;
(see include/linux/kprobes.h for more information on this).
/* pre_handler: this is called just before the probed instruction is |
You also need to specify the address of the kernel routine where you want to insert the probe during registration. Use any of these methods to get the kernel routine address:
- Get the address directly from the System.map file.
For example, to get the address of do_fork, execute$grep do_fork /usr/src/linux/System.map
at the command line. - Use the
nm
command.$nm vmlinuz |grep do_fork
- Obtain the address from the /proc/kallsyms file.
$cat /proc/kallsyms |grep do_fork
- Use the
kallsyms_lookup_name()
routine.
This routine is defined in the kernel/kallsyms.c file, and you must compile the kernel withCONFIG_KALLSYMS
enabled in order to use it.kallsyms_lookup_name()
takes a kernel routine name as a string and returns the address of that kernel routine. For example:kallsyms_lookup_name("do_fork");
Then register your probe in the init_module:
/* specify pre_handler address |
Once the probe is registered, running any shell command will result in a call
to do_fork
, and you will be able to see your printk's on the
console, or by running dmesg
. Remember to unregister the probe when
you are done:
unregister_kprobe(&kp);
The following output shows kprobe's address, and the contents of the eflags registers:
$tail -5 /var/log/messages
Jun 14 18:21:18 llm05 kernel:
pre_handler: p->addr=0xc01441d0, eflags=0x202
Jun 14 18:21:18 llm05
kernel: post_handler: p->addr=0xc01441d0, eflags=0x196
You can insert printk's at the beginning of a routine or at any offset in the function (the offset must be at the instruction boundary). The following code samples show how to calculate the offset. First, disassemble the machine instructions from the object file and save them as a file:
$objdump -D /usr/src/linux/kernel/fork.o > fork.dis
Which produces:
000022b0 |
To insert the probe at offset 0x22c4, get the relative offset from the
beginning of the routine 0x22c4 - 0x22b0 = 0x14
and then add the
offset to the address of do_fork 0xc01441d0 + 0x14
. (To ascertain
the address of do_fork, run $cat /proc/kallsyms | grep do_fork
.)
You can also add the relative offset of do_fork 0x22c4 - 0x22b0 =
0x14
to the output of kallsyms_lookup_name("do_fork");
Thus:
0x14 + kallsyms_lookup_name("do_fork");
Now, let's dump some elements of all of the jobs that are running on the system with a Kprobe post_handler that we've modified to dump data structures:
void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags) { |
This module should be inserted at the offset of do_fork.
$tail -10 /var/log/messages |
We already compiled in support for the SysRq key. Enable it with:
$echo 1 > /proc/sys/kernel/sysrq
Now you can use Alt+SysRq+W to view all inserted kernel probes on the console, or in /var/log/messages.
Jun 23 10:24:48 linux-udp4749545uds kernel: SysRq : Show kprobes |
|
Because probe event handlers run as extensions to the system breakpoint interrupt handler, they have little or no dependence on system facilities -- and so are able to be implanted in the most hostile environments, from interrupt-time, and task-time, to disabled, inter-context switch, and SMP-enabled code paths -- all without adversely skewing system performance.
The benefits of using Kprobes are many. printk's can be inserted without rebuilding and rebooting the kernel. Processor registers can be logged and even modified for debugging -- without disruption to the system. Similarly, Linux kernel data structures can also be logged and even modified non-disruptively, as well. You can even debug race conditions on SMP systems with Kprobes -- and save yourself the trouble of all that rebuilding and rebooting. You'll find kernel debugging is faster and easier than ever.
- Find more information, recent news, READMEs, and downloads at the Kprobes home page. The describes the kprobes interface in detail.
- Kprobes was developed from the full Dynamic Probes patch. Dynamic Probes use Kernel Hooks to gather difficult-to-acquire diagnostic
information.
- Support for Kprobes was merged into the kernel at v.2.5.26; see (Kernel Traffic, July
25 2002) for the announcement and a brief write-up. KProbes added in 2.5.73.
- Kprobes makes use of the BIOS interrupt key. This can be made into a key to defeat spyware, uncleanly reboot, show memory information, kill
processes, and more (much more). It's good for debugging; less good for production machines, where it can pose a
security threat.
- objdump displays information about one or more object files. For more
information, see the Linux 2.6 kernel modules.
- Captain's Universe has posted a document on .
- Download the source of the kernel ; this replaces modutils for
modern kernels. It's one of .
- Find more resources for Linux developers in the developerWorks Linux zone.
- Browse for books on these and other technical
topics.
- Develop and test your Linux applications using the latest IBM tools and
middleware with a developerWorks Subscription: you get IBM software from
WebSphere, DB2, Lotus, Rational, and Tivoli, and a license to use the software
for 12 months, all for less money than you might think.
- Download no-charge trial versions of selected developerWorks Subscription
products that run on Linux, including WebSphere Studio Site Developer, WebSphere
SDK for Web services, WebSphere Application Server, DB2 Universal Database
Personal Developers Edition, Tivoli Access Manager, and Lotus Domino Server,
from the Speed-start your Linux app section of developerWorks.
For an even speedier start, help yourself to a product-by-product collection of
how-to articles and tech support.
|
Prasanna S. Panchamukhi works as a developer for IBM's Linux Technology Center in Bangalore, India. He is currently involved in improving various debugging tools for Linux. Previously, he was involved in writing fiber channel device drivers and developing network processor applications, as well as maintaining Unix operating systems. You can reach Prasanna at . |