Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 1 | Demonstrations of filetop, the Linux eBPF/bcc version. |
| 2 | |
| 3 | |
| 4 | filetop shows reads and writes by file, with process details. For example: |
| 5 | |
| 6 | # ./filetop -C |
| 7 | Tracing... Output every 1 secs. Hit Ctrl-C to end |
| 8 | |
| 9 | 08:00:23 loadavg: 0.91 0.33 0.23 3/286 26635 |
| 10 | |
| 11 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 12 | 26628 ld 161 186 643 152 R built-in.o |
| 13 | 26634 cc1 1 0 200 0 R autoconf.h |
| 14 | 26618 cc1 1 0 200 0 R autoconf.h |
| 15 | 26634 cc1 12 0 192 0 R tracepoint.h |
| 16 | 26584 cc1 2 0 143 0 R mm.h |
| 17 | 26634 cc1 2 0 143 0 R mm.h |
| 18 | 26631 make 34 0 136 0 R auto.conf |
| 19 | 26634 cc1 1 0 98 0 R fs.h |
| 20 | 26584 cc1 1 0 98 0 R fs.h |
| 21 | 26634 cc1 1 0 91 0 R sched.h |
| 22 | 26634 cc1 1 0 78 0 R printk.c |
| 23 | 26634 cc1 3 0 73 0 R mmzone.h |
| 24 | 26628 ld 18 0 72 0 R hibernate.o |
| 25 | 26628 ld 16 0 64 0 R suspend.o |
| 26 | 26628 ld 16 0 64 0 R snapshot.o |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 27 | 26628 ld 16 0 64 0 R qos.o |
| 28 | 26628 ld 13 0 52 0 R main.o |
| 29 | 26628 ld 12 0 52 0 R swap.o |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 30 | [...] |
| 31 | |
Daniel Neiter | caa38c5 | 2017-03-01 17:21:25 -0800 | [diff] [blame] | 32 | This shows various files read and written during a Linux kernel build. By |
| 33 | default the output is sorted by the total read size in Kbytes (R_Kb). Sorting |
| 34 | order can be changed via -s option. This is instrumenting at the VFS interface, |
| 35 | so this is reads and writes that may return entirely from the file system cache |
| 36 | (page cache). |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 37 | |
| 38 | While not printed, the average read and write size can be calculated by |
| 39 | dividing R_Kb by READS, and the same for writes. |
| 40 | |
| 41 | The "T" column indicates the type of the file: "R" for regular files, "S" for |
Mark Drayton | 7434731 | 2016-08-25 20:46:35 +0100 | [diff] [blame] | 42 | sockets, and "O" for other (including pipes). By default only regular files are |
| 43 | shown; use the -a option to show all file types. |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 44 | |
| 45 | This script works by tracing the vfs_read() and vfs_write() functions using |
| 46 | kernel dynamic tracing, which instruments explicit read and write calls. If |
| 47 | files are read or written using another means (eg, via mmap()), then they |
| 48 | will not be visible using this tool. |
| 49 | |
| 50 | This should be useful for file system workload characterization when analyzing |
| 51 | the performance of applications. |
| 52 | |
| 53 | Note that tracing VFS level reads and writes can be a frequent activity, and |
| 54 | this tool can begin to cost measurable overhead at high I/O rates. |
| 55 | |
| 56 | |
| 57 | A -C option will stop clearing the screen, and -r with a number will restrict |
| 58 | the output to that many rows (20 by default). For example, not clearing |
| 59 | the screen and showing the top 5 only: |
| 60 | |
| 61 | # ./filetop -Cr 5 |
| 62 | Tracing... Output every 1 secs. Hit Ctrl-C to end |
| 63 | |
| 64 | 08:05:11 loadavg: 0.75 0.35 0.25 3/285 822 |
| 65 | |
| 66 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 67 | 32672 cksum 5006 0 320384 0 R data1 |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 68 | 809 run 2 0 8 0 R nsswitch.conf |
| 69 | 811 run 2 0 8 0 R nsswitch.conf |
| 70 | 804 chown 2 0 8 0 R nsswitch.conf |
| 71 | |
| 72 | 08:05:12 loadavg: 0.75 0.35 0.25 3/285 845 |
| 73 | |
| 74 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 75 | 32672 cksum 4986 0 319104 0 R data1 |
| 76 | 845 chown 2 0 8 0 R nsswitch.conf |
| 77 | 828 run 2 0 8 0 R nsswitch.conf |
| 78 | 835 run 2 0 8 0 R nsswitch.conf |
| 79 | 830 run 2 0 8 0 R nsswitch.conf |
| 80 | |
| 81 | 08:05:13 loadavg: 0.75 0.35 0.25 3/285 868 |
| 82 | |
| 83 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 84 | 32672 cksum 4985 0 319040 0 R data1 |
| 85 | 857 run 2 0 8 0 R nsswitch.conf |
| 86 | 858 run 2 0 8 0 R nsswitch.conf |
| 87 | 859 run 2 0 8 0 R nsswitch.conf |
| 88 | 848 run 2 0 8 0 R nsswitch.conf |
| 89 | [...] |
| 90 | |
Mark Drayton | 7434731 | 2016-08-25 20:46:35 +0100 | [diff] [blame] | 91 | This output shows a cksum command reading data1. |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 92 | |
| 93 | |
| 94 | An optional interval and optional count can also be added to the end of the |
| 95 | command line. For example, for 1 second interval, and 3 summaries in total: |
| 96 | |
Mark Drayton | 7434731 | 2016-08-25 20:46:35 +0100 | [diff] [blame] | 97 | # ./filetop -Cr 5 -a 1 3 |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 98 | Tracing... Output every 1 secs. Hit Ctrl-C to end |
| 99 | |
| 100 | 08:08:20 loadavg: 0.30 0.42 0.31 3/282 5187 |
| 101 | |
| 102 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 103 | 12421 sshd 14101 0 225616 0 O ptmx |
| 104 | 12296 sshd 4 0 64 0 O ptmx |
| 105 | 12421 sshd 3 14104 48 778 S TCP |
| 106 | 5178 run 2 0 8 0 R nsswitch.conf |
| 107 | 5165 run 2 0 8 0 R nsswitch.conf |
| 108 | |
| 109 | 08:08:21 loadavg: 0.30 0.42 0.31 5/282 5210 |
| 110 | |
| 111 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 112 | 12421 sshd 9159 0 146544 0 O ptmx |
| 113 | 12421 sshd 3 9161 48 534 S TCP |
| 114 | 12296 sshd 1 0 16 0 S TCP |
| 115 | 5188 run 2 0 8 0 R nsswitch.conf |
| 116 | 5203 run 2 0 8 0 R nsswitch.conf |
| 117 | |
| 118 | 08:08:22 loadavg: 0.30 0.42 0.31 2/282 5233 |
| 119 | |
| 120 | PID COMM READS WRITES R_Kb W_Kb T FILE |
| 121 | 12421 sshd 26166 0 418656 0 O ptmx |
| 122 | 12421 sshd 4 26171 64 1385 S TCP |
| 123 | 12296 sshd 1 0 16 0 O ptmx |
| 124 | 5214 run 2 0 8 0 R nsswitch.conf |
| 125 | 5227 run 2 0 8 0 R nsswitch.conf |
| 126 | Detaching... |
| 127 | |
Mark Drayton | 7434731 | 2016-08-25 20:46:35 +0100 | [diff] [blame] | 128 | This example shows the -a option to include all file types. It caught heavy |
| 129 | socket I/O from an sshd process, showing up as non-regular file types (the "O" |
| 130 | for other, and "S" for socket, in the type column: "T"). |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 131 | |
| 132 | |
| 133 | USAGE message: |
| 134 | |
| 135 | # ./filetop -h |
Mark Drayton | 7434731 | 2016-08-25 20:46:35 +0100 | [diff] [blame] | 136 | usage: filetop.py [-h] [-a] [-C] [-r MAXROWS] [-p PID] [interval] [count] |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 137 | |
| 138 | File reads and writes by process |
| 139 | |
| 140 | positional arguments: |
| 141 | interval output interval, in seconds |
| 142 | count number of outputs |
| 143 | |
| 144 | optional arguments: |
| 145 | -h, --help show this help message and exit |
Mark Drayton | 7434731 | 2016-08-25 20:46:35 +0100 | [diff] [blame] | 146 | -a, --all-files include non-regular file types (sockets, FIFOs, etc) |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 147 | -C, --noclear don't clear the screen |
| 148 | -r MAXROWS, --maxrows MAXROWS |
| 149 | maximum rows to print, default 20 |
Daniel Neiter | caa38c5 | 2017-03-01 17:21:25 -0800 | [diff] [blame] | 150 | -s {reads,writes,rbytes,wbytes}, --sort {reads,writes,rbytes,wbytes} |
| 151 | sort column, default rbytes |
Brendan Gregg | 08c2981 | 2016-02-09 00:36:43 -0800 | [diff] [blame] | 152 | -p PID, --pid PID trace this PID only |
| 153 | |
| 154 | examples: |
| 155 | ./filetop # file I/O top, 1 second refresh |
| 156 | ./filetop -C # don't clear the screen |
| 157 | ./filetop -p 181 # PID 181 only |
| 158 | ./filetop 5 # 5 second summaries |
| 159 | ./filetop 5 10 # 5 second summaries, 10 times only |