Sasha Goldshtein | 3e39a08 | 2016-03-24 08:39:47 -0700 | [diff] [blame] | 1 | Demonstrations of tplist. |
| 2 | |
| 3 | |
| 4 | tplist displays kernel tracepoints and USDT probes, including their |
| 5 | format. It can be used to discover probe points for use with the trace |
| 6 | and argdist tools. Kernel tracepoints are scattered around the kernel |
| 7 | and provide valuable static tracing on block and network I/O, scheduling, |
| 8 | power events, and many other subjects. USDT probes are placed in libraries |
| 9 | (such as libc) and executables (such as node) and provide static tracing |
| 10 | information that can (optionally) be turned on and off at runtime. |
| 11 | |
| 12 | For example, suppose you want to discover which USDT probes a particular |
| 13 | executable contains. Just run tplist on that executable (or library): |
| 14 | |
| 15 | $ tplist -l basic_usdt |
| 16 | /home/vagrant/basic_usdt basic_usdt:start_main |
| 17 | /home/vagrant/basic_usdt basic_usdt:loop_iter |
| 18 | /home/vagrant/basic_usdt basic_usdt:end_main |
| 19 | |
Sasha Goldshtein | 69e361a | 2016-09-27 19:40:00 +0300 | [diff] [blame] | 20 | The loop_iter probe sounds interesting. How many arguments are available? |
Sasha Goldshtein | 3e39a08 | 2016-03-24 08:39:47 -0700 | [diff] [blame] | 21 | |
| 22 | $ tplist '*loop_iter' -l basic_usdt -v |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 23 | basic_usdt:loop_iter [sema 0x601036] |
Sasha Goldshtein | 69e361a | 2016-09-27 19:40:00 +0300 | [diff] [blame] | 24 | 2 location(s) |
| 25 | 2 argument(s) |
Sasha Goldshtein | 3e39a08 | 2016-03-24 08:39:47 -0700 | [diff] [blame] | 26 | |
| 27 | This output indicates that the loop_iter probe is used in two locations |
Sasha Goldshtein | 69e361a | 2016-09-27 19:40:00 +0300 | [diff] [blame] | 28 | in the basic_usdt executable, and that it has two arguments. Fortunately, |
| 29 | the argdist and trace tools understand the probe format and can print out |
| 30 | the arguments automatically -- you can refer to them as arg1, arg2, and |
| 31 | so on. |
Sasha Goldshtein | 3e39a08 | 2016-03-24 08:39:47 -0700 | [diff] [blame] | 32 | |
| 33 | Try to explore with some common libraries on your system and see if they |
| 34 | contain UDST probes. Here are two examples you might find interesting: |
| 35 | |
| 36 | $ tplist -l pthread # list probes in libpthread |
| 37 | /lib64/libpthread.so.0 libpthread:pthread_start |
| 38 | /lib64/libpthread.so.0 libpthread:pthread_create |
| 39 | /lib64/libpthread.so.0 libpthread:pthread_join |
| 40 | /lib64/libpthread.so.0 libpthread:pthread_join_ret |
| 41 | /lib64/libpthread.so.0 libpthread:mutex_init |
| 42 | ... more output truncated |
| 43 | |
| 44 | $ tplist -l c # list probes in libc |
| 45 | /lib64/libc.so.6 libc:setjmp |
| 46 | /lib64/libc.so.6 libc:longjmp |
| 47 | /lib64/libc.so.6 libc:longjmp_target |
| 48 | /lib64/libc.so.6 libc:memory_arena_reuse_free_list |
| 49 | /lib64/libc.so.6 libc:memory_heap_new |
| 50 | ... more output truncated |
| 51 | |
| 52 | tplist also understands kernel tracepoints, and can list their format |
| 53 | as well. For example, let's look for all block I/O-related tracepoints: |
| 54 | |
| 55 | # tplist 'block*' |
| 56 | block:block_touch_buffer |
| 57 | block:block_dirty_buffer |
| 58 | block:block_rq_abort |
| 59 | block:block_rq_requeue |
| 60 | block:block_rq_complete |
| 61 | block:block_rq_insert |
| 62 | block:block_rq_issue |
| 63 | block:block_bio_bounce |
| 64 | block:block_bio_complete |
| 65 | block:block_bio_backmerge |
| 66 | block:block_bio_frontmerge |
| 67 | block:block_bio_queue |
| 68 | block:block_getrq |
| 69 | block:block_sleeprq |
| 70 | block:block_plug |
| 71 | block:block_unplug |
| 72 | block:block_split |
| 73 | block:block_bio_remap |
| 74 | block:block_rq_remap |
| 75 | |
| 76 | The block:block_rq_complete tracepoints sounds interesting. Let's print |
| 77 | its format to see what we can trace with argdist and trace: |
| 78 | |
| 79 | $ tplist -v block:block_rq_complete |
| 80 | block:block_rq_complete |
| 81 | dev_t dev; |
| 82 | sector_t sector; |
| 83 | unsigned int nr_sector; |
| 84 | int errors; |
| 85 | char rwbs[8]; |
| 86 | |
| 87 | The dev, sector, nr_sector, etc. variables can now all be used in probes |
| 88 | you specify with argdist or trace. |
| 89 | |
| 90 | |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 91 | For debugging USDT probes, it is sometimes useful to see the exact locations |
| 92 | and arguments of the probes, including the registers or global variables from |
| 93 | which their values are coming from. In super-verbose mode, tplist will print |
| 94 | this information (note the -vv): |
| 95 | |
| 96 | $ tplist -vv -l c *alloc* |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 97 | libc:memory_malloc_retry [sema 0x0] |
| 98 | location #0 /lib64/libc.so.6 0x835c0 |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 99 | argument #0 8 unsigned bytes @ bp |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 100 | location #1 /lib64/libc.so.6 0x83778 |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 101 | argument #0 8 unsigned bytes @ bp |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 102 | location #2 /lib64/libc.so.6 0x85a50 |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 103 | argument #0 8 unsigned bytes @ bp |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 104 | libc:memory_realloc_retry [sema 0x0] |
| 105 | location #0 /lib64/libc.so.6 0x84b90 |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 106 | argument #0 8 unsigned bytes @ r13 |
| 107 | argument #1 8 unsigned bytes @ bp |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 108 | location #1 /lib64/libc.so.6 0x85cf0 |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 109 | argument #0 8 unsigned bytes @ r13 |
| 110 | argument #1 8 unsigned bytes @ bp |
Yonghong Song | 2489457 | 2018-01-04 22:08:51 -0800 | [diff] [blame] | 111 | libc:memory_calloc_retry [sema 0x0] |
| 112 | location #0 /lib64/libc.so.6 0x850f0 |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 113 | argument #0 8 unsigned bytes @ bp |
| 114 | |
| 115 | |
Sasha Goldshtein | 3e39a08 | 2016-03-24 08:39:47 -0700 | [diff] [blame] | 116 | USAGE message: |
| 117 | |
| 118 | $ tplist -h |
| 119 | usage: tplist.py [-h] [-p PID] [-l LIB] [-v] [filter] |
| 120 | |
| 121 | Display kernel tracepoints or USDT probes and their formats. |
| 122 | |
| 123 | positional arguments: |
| 124 | filter A filter that specifies which probes/tracepoints to print |
| 125 | |
| 126 | optional arguments: |
| 127 | -h, --help show this help message and exit |
| 128 | -p PID, --pid PID List USDT probes in the specified process |
| 129 | -l LIB, --lib LIB List USDT probes in the specified library or executable |
Sasha Goldshtein | 6e91a74 | 2016-10-06 18:18:18 +0300 | [diff] [blame] | 130 | -v Increase verbosity level (print variables, arguments, etc.) |
Sasha Goldshtein | 3e39a08 | 2016-03-24 08:39:47 -0700 | [diff] [blame] | 131 | |