-m, --module=MODNAME|PATH Specify module name in which perf-probe searches probe points or lines. If a path of module file is passed, perf-probe treat it as an offline module (this means you can add a probe on a module which has not been loaded yet). |
如下例子来验证这个feature
从记录中看到,系统没又加载vxlan module;
接着新增一个动态event,可以探测vxlan_xmit的第85行;
perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko -a vxlan_xmit:85
注册成功,可以通过perf probe -l查看结果。
root@jianlin-100:/home/jianlin# lsmod |grep -i vxlan root@jianlin-100:/home/jianlin# root@jianlin-100:/home/jianlin# perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko -a vxlan_xmit:85 Added new event: probe:vxlan_xmit_L85 (on vxlan_xmit:85 in vxlan) You can now use it in all perf tools, such as: perf record -e probe:vxlan_xmit_L85 -aR sleep 1 root@jianlin-100:/home/jianlin# perf probe -l probe:vxlan_xmit_L85 (on vxlan_xmit+1596 in vxlan) |
但是对于unload module的kretprobe事件,注册并不会成功
root@jianlin-100:/home/jianlin# lsmod |grep -i vxlan root@jianlin-100:/home/jianlin# root@jianlin-100:/home/jianlin# perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko 'vxlan_xmit%return $retval' Failed to write event: Invalid argument Error: Failed to add events. root@jianlin-100:/home/jianlin# root@jianlin-100:/home/jianlin# cat /sys/kernel/debug/tracing/error_log [169096.405945] trace_kprobe: error: Failed to register probe event Command: r:probe/vxlan_xmit__return vxlan:vxlan_xmit+0 $retval ^ |
对于已经加载了的module,kretprobe不会出问题;
root@jianlin-100:/home/jianlin# modprobe vxlan root@jianlin-100:/home/jianlin# root@jianlin-100:/home/jianlin# perf probe -m /lib/modules/5.11.0-rc2+/kernel/drivers/net/vxlan.ko 'vxlan_xmit%return $retval' Added new event: probe:vxlan_xmit__return (on vxlan_xmit%return in vxlan with $retval) You can now use it in all perf tools, such as: perf record -e probe:vxlan_xmit__return -aR sleep 1 root@jianlin-100:/home/jianlin# root@jianlin-100:/home/jianlin# lsmod |grep -i vxlan vxlan 77824 0 ip6_udp_tunnel 16384 1 vxlan udp_tunnel 24576 1 vxlan |
该问题已经report给kernel community,参见:
问题原因就是 在检查kernel symbol时,由于offline kernel module未加载,所以kprobe_on_func_entry总是返回失败,__register_trace_kprobe返回值为-EINVAL; 而此时代码逻辑是ret为-ENOENT,才会继续判断是否要保留module的event还是需要unregister.
所以kretprobe对离线module无效.
If register_kretprobe()returns an error -EINVAL. This means that __register_trace_kprobe return -EINVAL, --- ret = __register_trace_kprobe(tk); if (ret == -ENOENT && !trace_kprobe_module_exist(tk)) { pr_warn("This probe might be able to register after target module is loaded. Continue.\n"); ret = 0; } --- As code show, cannot enable kretprobe for an unloaded module. |
这个问题已经修复,请见:
https://lore.kernel.org/lkml/161176187132.1067016.8118042342894378981.stgit@devnote2/
该问题的引入应该是:
Fixes: 59158ec4aef7 ("tracing/kprobes: Check the probe on unloaded module correctly")