| Kprobe-based Event Tracer |
| ========================= |
| |
| Documentation is written by Masami Hiramatsu |
| |
| |
| Overview |
| -------- |
| This tracer is similar to the events tracer which is based on Tracepoint |
| infrastructure. Instead of Tracepoint, this tracer is based on kprobes(kprobe |
| and kretprobe). It probes anywhere where kprobes can probe(this means, all |
| functions body except for __kprobes functions). |
| |
| Unlike the function tracer, this tracer can probe instructions inside of |
| kernel functions. It allows you to check which instruction has been executed. |
| |
| Unlike the Tracepoint based events tracer, this tracer can add and remove |
| probe points on the fly. |
| |
| Similar to the events tracer, this tracer doesn't need to be activated via |
| current_tracer, instead of that, just set probe points via |
| /sys/kernel/debug/tracing/kprobe_events. And you can set filters on each |
| probe events via /sys/kernel/debug/tracing/events/kprobes/<EVENT>/filter. |
| |
| |
| Synopsis of kprobe_events |
| ------------------------- |
| p[:EVENT] SYMBOL[+offs|-offs]|MEMADDR [FETCHARGS] : Set a probe |
| r[:EVENT] SYMBOL[+0] [FETCHARGS] : Set a return probe |
| |
| EVENT : Event name. If omitted, the event name is generated |
| based on SYMBOL+offs or MEMADDR. |
| SYMBOL[+offs|-offs] : Symbol+offset where the probe is inserted. |
| MEMADDR : Address where the probe is inserted. |
| |
| FETCHARGS : Arguments. Each probe can have up to 128 args. |
| %REG : Fetch register REG |
| sN : Fetch Nth entry of stack (N >= 0) |
| sa : Fetch stack address. |
| @ADDR : Fetch memory at ADDR (ADDR should be in kernel) |
| @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol) |
| aN : Fetch function argument. (N >= 0)(*) |
| rv : Fetch return value.(**) |
| ra : Fetch return address.(**) |
| +|-offs(FETCHARG) : fetch memory at FETCHARG +|- offs address.(***) |
| |
| (*) aN may not correct on asmlinkaged functions and at the middle of |
| function body. |
| (**) only for return probe. |
| (***) this is useful for fetching a field of data structures. |
| |
| |
| Per-Probe Event Filtering |
| ------------------------- |
| Per-probe event filtering feature allows you to set different filter on each |
| probe and gives you what arguments will be shown in trace buffer. If an event |
| name is specified right after 'p:' or 'r:' in kprobe_events, the tracer adds |
| an event under tracing/events/kprobes/<EVENT>, at the directory you can see |
| 'id', 'enabled', 'format' and 'filter'. |
| |
| enabled: |
| You can enable/disable the probe by writing 1 or 0 on it. |
| |
| format: |
| It shows the format of this probe event. It also shows aliases of arguments |
| which you specified to kprobe_events. |
| |
| filter: |
| You can write filtering rules of this event. And you can use both of aliase |
| names and field names for describing filters. |
| |
| |
| Event Profiling |
| --------------- |
| You can check the total number of probe hits and probe miss-hits via |
| /sys/kernel/debug/tracing/kprobe_profile. |
| The first column is event name, the second is the number of probe hits, |
| the third is the number of probe miss-hits. |
| |
| |
| Usage examples |
| -------------- |
| To add a probe as a new event, write a new definition to kprobe_events |
| as below. |
| |
| echo p:myprobe do_sys_open a0 a1 a2 a3 > /sys/kernel/debug/tracing/kprobe_events |
| |
| This sets a kprobe on the top of do_sys_open() function with recording |
| 1st to 4th arguments as "myprobe" event. |
| |
| echo r:myretprobe do_sys_open rv ra >> /sys/kernel/debug/tracing/kprobe_events |
| |
| This sets a kretprobe on the return point of do_sys_open() function with |
| recording return value and return address as "myretprobe" event. |
| You can see the format of these events via |
| /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format. |
| |
| cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format |
| name: myprobe |
| ID: 23 |
| format: |
| field:unsigned short common_type; offset:0; size:2; |
| field:unsigned char common_flags; offset:2; size:1; |
| field:unsigned char common_preempt_count; offset:3; size:1; |
| field:int common_pid; offset:4; size:4; |
| field:int common_tgid; offset:8; size:4; |
| |
| field: unsigned long ip; offset:16;tsize:8; |
| field: int nargs; offset:24;tsize:4; |
| field: unsigned long arg0; offset:32;tsize:8; |
| field: unsigned long arg1; offset:40;tsize:8; |
| field: unsigned long arg2; offset:48;tsize:8; |
| field: unsigned long arg3; offset:56;tsize:8; |
| |
| alias: a0; original: arg0; |
| alias: a1; original: arg1; |
| alias: a2; original: arg2; |
| alias: a3; original: arg3; |
| |
| print fmt: "%lx: 0x%lx 0x%lx 0x%lx 0x%lx", ip, arg0, arg1, arg2, arg3 |
| |
| |
| You can see that the event has 4 arguments and alias expressions |
| corresponding to it. |
| |
| echo > /sys/kernel/debug/tracing/kprobe_events |
| |
| This clears all probe points. and you can see the traced information via |
| /sys/kernel/debug/tracing/trace. |
| |
| cat /sys/kernel/debug/tracing/trace |
| # tracer: nop |
| # |
| # TASK-PID CPU# TIMESTAMP FUNCTION |
| # | | | | | |
| <...>-1447 [001] 1038282.286875: do_sys_open+0x0/0xd6: 0x3 0x7fffd1ec4440 0x8000 0x0 |
| <...>-1447 [001] 1038282.286878: sys_openat+0xc/0xe <- do_sys_open: 0xfffffffffffffffe 0xffffffff81367a3a |
| <...>-1447 [001] 1038282.286885: do_sys_open+0x0/0xd6: 0xffffff9c 0x40413c 0x8000 0x1b6 |
| <...>-1447 [001] 1038282.286915: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a |
| <...>-1447 [001] 1038282.286969: do_sys_open+0x0/0xd6: 0xffffff9c 0x4041c6 0x98800 0x10 |
| <...>-1447 [001] 1038282.286976: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a |
| |
| |
| Each line shows when the kernel hits a probe, and <- SYMBOL means kernel |
| returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel |
| returns from do_sys_open to sys_open+0x1b). |
| |
| |