kretprobe: 关于offline module的一个bug

780阅读 0评论2021-01-28 静默梧桐
分类:LINUX

perf-probe 支持对unload kernel module添加dynamic event,待module被加载后可以使能对应的event;其help中 -m option说明了这一点:
       -m, --module=MODNAME|PATH
           Specify module name in which perf-probe searches probe points or lines. If a path of module file is passed, perf-probe treat it as an offline module (this means you can add a probe on a module which has not been loaded yet).

如下例子来验证这个feature
从记录中看到,系统没又加载vxlan module;
接着新增一个动态event,可以探测vxlan_xmit的第85行;
    perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko  -a  vxlan_xmit:85
注册成功,可以通过perf probe -l查看结果。


root@jianlin-100:/home/jianlin# lsmod  |grep -i vxlan
root@jianlin-100:/home/jianlin#
root@jianlin-100:/home/jianlin# perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko  -a  vxlan_xmit:85
Added new event:
  probe:vxlan_xmit_L85 (on vxlan_xmit:85 in vxlan)

You can now use it in all perf tools, such as:

        perf record -e probe:vxlan_xmit_L85 -aR sleep 1

root@jianlin-100:/home/jianlin# perf probe -l
  probe:vxlan_xmit_L85 (on vxlan_xmit+1596 in vxlan)


但是对于unload module的kretprobe事件,注册并不会成功

root@jianlin-100:/home/jianlin# lsmod |grep -i vxlan
root@jianlin-100:/home/jianlin#
root@jianlin-100:/home/jianlin# perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko  'vxlan_xmit%return $retval'
Failed to write event: Invalid argument
  Error: Failed to add events.
root@jianlin-100:/home/jianlin#
root@jianlin-100:/home/jianlin# cat /sys/kernel/debug/tracing/error_log
[169096.405945] trace_kprobe: error: Failed to register probe event
  Command: r:probe/vxlan_xmit__return vxlan:vxlan_xmit+0 $retval
                                      ^


对于已经加载了的module,kretprobe不会出问题;
root@jianlin-100:/home/jianlin# modprobe vxlan
root@jianlin-100:/home/jianlin#
root@jianlin-100:/home/jianlin# perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko  'vxlan_xmit%return $retval'
Added new event:
  probe:vxlan_xmit__return (on vxlan_xmit%return in vxlan with $retval)

You can now use it in all perf tools, such as:

        perf record -e probe:vxlan_xmit__return -aR sleep 1

root@jianlin-100:/home/jianlin#
root@jianlin-100:/home/jianlin# lsmod  |grep -i vxlan
vxlan                  77824  0
ip6_udp_tunnel         16384  1 vxlan
udp_tunnel             24576  1 vxlan


该问题已经report给kernel community,参见:


问题原因就是 在检查kernel symbol时,由于offline kernel module未加载,所以kprobe_on_func_entry总是返回失败,__register_trace_kprobe返回值为-EINVAL; 而此时代码逻辑是ret为-ENOENT,才会继续判断是否要保留module的event还是需要unregister.
所以kretprobe对离线module无效.
If register_kretprobe()returns an error -EINVAL.
This means that __register_trace_kprobe return -EINVAL,

---
ret = __register_trace_kprobe(tk);
if (ret == -ENOENT && !trace_kprobe_module_exist(tk)) {
pr_warn("This probe might be able to register after target module is loaded. Continue.\n");
ret = 0;
}
---
As code show, cannot enable kretprobe for an unloaded module.

这个问题已经修复,请见:
https://lore.kernel.org/lkml/161176187132.1067016.8118042342894378981.stgit@devnote2/

该问题的引入应该是:
Fixes: 59158ec4aef7 ("tracing/kprobes: Check the probe on unloaded module correctly")
这个问题是在与kernel maintainer讨论我提交的一个patch时发现的,bug由我发现并report,但我提交的patch没有办法继续merge了,失去了一次kernel留名的机会。




上一篇:GCC:关于-fpatchable-function-entry的一个issue
下一篇:Arm64 FP/LR寄存器解析