| .TH argdist 8 "2016-02-11" "USER COMMANDS" |
| .SH NAME |
| argdist \- Trace a function and display a histogram or frequency count of its parameter values. Uses Linux eBPF/bcc. |
| .SH SYNOPSIS |
| .B argdist [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-d DURATION] [-n COUNT] [-v] [-T TOP] [-H specifier] [-C specifier] [-I header] |
| .SH DESCRIPTION |
| argdist attaches to function entry and exit points, collects specified parameter |
| values, and stores them in a histogram or a frequency collection that counts |
| the number of times a parameter value occurred. It can also filter parameter |
| values and instrument multiple entry points at once. |
| |
| Since this uses BPF, only the root user can use this tool. |
| .SH REQUIREMENTS |
| CONFIG_BPF and bcc. |
| .SH OPTIONS |
| .TP |
| \-h |
| Print usage message. |
| .TP |
| \-p PID |
| Trace only functions in the process PID. |
| .TP |
| \-z STRING_SIZE |
| When collecting string arguments (of type char*), collect up to STRING_SIZE |
| characters. Longer strings will be truncated. |
| .TP |
| \-i INTERVAL |
| Print the collected data every INTERVAL seconds. The default is 1 second. |
| .TP |
| \-d DURATION |
| Total duration of trace in seconds. |
| .TP |
| \-n NUMBER |
| Print the collected data COUNT times and then exit. |
| .TP |
| \-v |
| Display the generated BPF program, for debugging purposes. |
| .TP |
| \-T TOP |
| When collecting frequency counts, display only the top TOP entries. |
| .TP |
| \-H specifiers, \-C specifiers |
| One or more probe specifications that instruct argdist which functions to |
| probe, which parameters to collect, how to aggregate them, and whether to perform |
| any filtering. See SPECIFIER SYNTAX below. |
| .TP |
| \-I header |
| One or more header files that should be included in the BPF program. This |
| enables the use of structure definitions, enumerations, and constants that |
| are available in these headers. You should provide the same path you would |
| include in the BPF program, e.g. 'linux/blkdev.h' or 'linux/time.h'. Note: in |
| many cases, argdist will deduce the necessary header files automatically. |
| .SH SPECIFIER SYNTAX |
| The general specifier syntax is as follows: |
| |
| .B {p,r,t,u}:{[library],category}:function(signature)[:type[,type...]:expr[,expr...][:filter]][#label] |
| .TP |
| .B {p,r,t,u} |
| Probe type \- "p" for function entry, "r" for function return, "t" for kernel |
| tracepoint, "u" for USDT probe; \-H for histogram collection, \-C for frequency count. |
| Indicates where to place the probe and whether the probe should collect frequency |
| count information, or aggregate the collected values into a histogram. Counting |
| probes will collect the number of times every parameter value was observed, |
| whereas histogram probes will collect the parameter values into a histogram. |
| Only integral types can be used with histogram probes; there is no such limitation |
| for counting probes. |
| .TP |
| .B [library] |
| Library containing the probe. |
| Specify the full path to the .so or executable file where the function to probe |
| resides. Alternatively, you can specify just the lib name: for example, "c" |
| refers to libc. If no library name is specified, the kernel is assumed. |
| .TP |
| .B category |
| The category of the kernel tracepoint. For example: net, sched, block. |
| .TP |
| .B function(signature) |
| The function to probe, and its signature. |
| The function name must match exactly for the probe to be placed. The signature, |
| on the other hand, is only required if you plan to collect parameter values |
| based on that signature. For example, if you only want to collect the first |
| parameter, you don't have to specify the rest of the parameters in the signature. |
| When capturing kernel tracepoints, this should be the name of the event, e.g. |
| net_dev_start_xmit. The signature for kernel tracepoints should be empty. When |
| capturing USDT probes, this should be the name of the probe, e.g. reloc_complete. |
| The signature for USDT probes should be empty. |
| .TP |
| .B [type[,type...]] |
| The type(s) of the expression(s) to capture. |
| This is the type of the keys in the histogram or raw event collection that are |
| collected by the probes. |
| .TP |
| .B [expr[,expr...]] |
| The expression(s) to capture. |
| These are the values that are assigned to the histogram or raw event collection. |
| You may use the parameters directly, or valid C expressions that involve the |
| parameters, such as "size % 10". |
| Tracepoints may access a special structure called "args" that is formatted |
| according to the tracepoint format (which you can obtain using tplist). |
| For example, the block:block_rq_complete tracepoint can access args->nr_sector. |
| USDT probes may access the arguments defined by the tracing program in the |
| special arg1, arg2, ... variables. To obtain their types, use the tplist tool. |
| Return probes can use the argument values received by the |
| function when it was entered, through the $entry(paramname) special variable. |
| Return probes can also access the function's return value in $retval, and the |
| function's execution time in nanoseconds in $latency. Note that adding the |
| $latency or $entry(paramname) variables to the expression will introduce an |
| additional probe at the function's entry to collect this data, and therefore |
| introduce additional overhead. |
| .TP |
| .B [filter] |
| The filter applied to the captured data. |
| Only parameter values that pass the filter will be collected. This is any valid |
| C expression that refers to the parameter values, such as "fd == 1 && length > 16". |
| The $entry, $retval, and $latency variables can be used here as well, in return |
| probes. |
| The filter expression may also use the STRCMP pseudo-function to compare |
| a predefined string to a string argument. For example: STRCMP("test.txt", file). |
| The order of arguments is important: the first argument MUST be a quoted |
| literal string, and the second argument can be a runtime string. |
| .TP |
| .B [label] |
| The label that will be displayed when printing the probed values. By default, |
| this is the probe specifier. |
| .SH EXAMPLES |
| .TP |
| Print a histogram of allocation sizes passed to kmalloc: |
| # |
| .B argdist -H 'p::__kmalloc(u64 size):u64:size' |
| .TP |
| Print a count of how many times process 1005 called malloc with an allocation size of 16 bytes: |
| # |
| .B argdist -p 1005 -C 'p:c:malloc(size_t size):size_t:size:size==16' |
| .TP |
| Snoop on all strings returned by gets(): |
| # |
| .B argdist -C 'r:c:gets():char*:$retval' |
| .TP |
| Print a histogram of read sizes that were longer than 1ms: |
| # |
| .B argdist -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t:$entry(count):$latency > 1000000' |
| .TP |
| Print frequency counts of how many times writes were issued to a particular file descriptor number, in process 1005: |
| # |
| .B argdist -p 1005 -C 'p:c:write(int fd):int:fd' |
| .TP |
| Print a histogram of error codes returned by read() in process 1005: |
| # |
| .B argdist -p 1005 -H 'r:c:read()' |
| .TP |
| Print a histogram of buffer sizes passed to write() across all processes, where the file descriptor was 1 (STDOUT): |
| # |
| .B argdist -H 'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1' |
| .TP |
| Count fork() calls in libc across all processes, grouped by pid: |
| # |
| .B argdist -C 'p:c:fork():int:$PID;fork per process' |
| .TP |
| Print histogram of number of sectors in completing block I/O requests: |
| # |
| .B argdist -H 't:block:block_rq_complete():u32:nr_sector' |
| .TP |
| Aggregate interrupts by interrupt request (IRQ): |
| # |
| .B argdist -C 't:irq:irq_handler_entry():int:irq' |
| .TP |
| Print the functions used as thread entry points and how common they are: |
| # |
| .B argdist -C 'u:pthread:pthread_start():u64:arg2' -p 1337 |
| .TP |
| Print histograms of sleep() and nanosleep() parameter values: |
| # |
| .B argdist -H 'p:c:sleep(u32 seconds):u32:seconds' -H 'p:c:nanosleep(struct timespec *req):long:req->tv_nsec' |
| .TP |
| Spy on writes to STDOUT performed by process 2780, up to a string size of 120 characters: |
| # |
| .B argdist -p 2780 -z 120 -C 'p:c:write(int fd, char* buf, size_t len):char*:buf:fd==1' |
| .TP |
| Group files being read from and the read sizes from __vfs_read: |
| # |
| .B argdist -C 'p::__vfs_read(struct file *file, void *buf, size_t count):char*,size_t:file->f_path.dentry->d_iname,count:file->f_path.dentry->d_iname[0]!=0' |
| .SH SOURCE |
| This is from bcc. |
| .IP |
| https://github.com/iovisor/bcc |
| .PP |
| Also look in the bcc distribution for a companion _examples.txt file containing |
| example usage, output, and commentary for this tool. |
| .SH OS |
| Linux |
| .SH STABILITY |
| Unstable - in development. |
| .SH AUTHOR |
| Sasha Goldshtein |