| Brendan Gregg | 6f075b9 | 2016-02-07 00:46:34 -0800 | [diff] [blame] | 1 | Demonstrations of biotop, the Linux eBPF/bcc version. |
| 2 | |
| 3 | |
| 4 | Short for block device I/O top, biotop summarizes which processes are |
| 5 | performing disk I/O. It's top for disks. Sample output: |
| 6 | |
| 7 | # ./biotop |
| 8 | Tracing... Output every 1 secs. Hit Ctrl-C to end |
| 9 | |
| 10 | 08:04:11 loadavg: 1.48 0.87 0.45 1/287 14547 |
| 11 | |
| 12 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 13 | 14501 cksum R 202 1 xvda1 361 28832 3.39 |
| 14 | 6961 dd R 202 1 xvda1 1628 13024 0.59 |
| 15 | 13855 dd R 202 1 xvda1 1627 13016 0.59 |
| 16 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.00 |
| 17 | 1880 supervise W 202 1 xvda1 2 8 6.71 |
| 18 | 1873 supervise W 202 1 xvda1 2 8 2.51 |
| 19 | 1871 supervise W 202 1 xvda1 2 8 1.57 |
| 20 | 1876 supervise W 202 1 xvda1 2 8 1.22 |
| 21 | 1892 supervise W 202 1 xvda1 2 8 0.62 |
| 22 | 1878 supervise W 202 1 xvda1 2 8 0.78 |
| 23 | 1886 supervise W 202 1 xvda1 2 8 1.30 |
| 24 | 1894 supervise W 202 1 xvda1 2 8 3.46 |
| 25 | 1869 supervise W 202 1 xvda1 2 8 0.73 |
| 26 | 1888 supervise W 202 1 xvda1 2 8 1.48 |
| 27 | |
| 28 | By default the screen refreshes every 1 second, and shows the top 20 disk |
| 29 | consumers, sorted on total Kbytes. The first line printed is the header, |
| 30 | which has the time and then the contents of /proc/loadavg. |
| 31 | |
| 32 | For the interval summarized by the output above, the "cksum" command performed |
| 33 | 361 disk reads to the "xvda1" device, for a total of 28832 Kbytes, with an |
| 34 | average I/O time of 3.39 ms. Two "dd" processes were also reading from the |
| 35 | same disk, which a higher I/O rate and lower latency. While the average I/O |
| 36 | size is not printed, it can be determined by dividing the Kbytes column by |
| 37 | the I/O column. |
| 38 | |
| 39 | The columns through to Kbytes show the workload applied. The final column, |
| 40 | AVGms, shows resulting performance. Other bcc tools can be used to get more |
| 41 | details when needed: biolatency and biosnoop. |
| 42 | |
| 43 | Many years ago I created the original "iotop", and later regretted not calling |
| 44 | it diskiotop or blockiotop, as "io" alone is ambiguous. This time it is biotop. |
| 45 | |
| 46 | |
| 47 | The -C option can be used to prevent the screen from clearing (my preference). |
| 48 | Here's using it with a 5 second interval: |
| 49 | |
| 50 | # ./biotop -C 5 |
| 51 | Tracing... Output every 5 secs. Hit Ctrl-C to end |
| 52 | |
| 53 | 08:09:44 loadavg: 0.42 0.44 0.39 2/282 22115 |
| 54 | |
| 55 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 56 | 22069 dd R 202 1 xvda1 5993 47976 0.33 |
| 57 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.67 |
| 58 | 1866 svscan R 202 1 xvda1 33 132 1.24 |
| 59 | 1880 supervise W 202 1 xvda1 10 40 0.56 |
| 60 | 1873 supervise W 202 1 xvda1 10 40 0.79 |
| 61 | 1871 supervise W 202 1 xvda1 10 40 0.78 |
| 62 | 1876 supervise W 202 1 xvda1 10 40 0.68 |
| 63 | 1892 supervise W 202 1 xvda1 10 40 0.71 |
| 64 | 1878 supervise W 202 1 xvda1 10 40 0.65 |
| 65 | 1886 supervise W 202 1 xvda1 10 40 0.78 |
| 66 | 1894 supervise W 202 1 xvda1 10 40 0.80 |
| 67 | 1869 supervise W 202 1 xvda1 10 40 0.91 |
| 68 | 1888 supervise W 202 1 xvda1 10 40 0.63 |
| 69 | 22069 bash R 202 1 xvda1 1 16 19.94 |
| 70 | 9251 kworker/u16:2 W 202 16 xvdb 2 8 0.13 |
| 71 | |
| 72 | 08:09:49 loadavg: 0.47 0.44 0.39 1/282 22231 |
| 73 | |
| 74 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 75 | 22069 dd R 202 1 xvda1 13450 107600 0.35 |
| 76 | 22199 cksum R 202 1 xvda1 941 45548 4.63 |
| 77 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.93 |
| 78 | 24467 kworker/0:2 W 202 16 xvdb 1 64 0.28 |
| 79 | 1880 supervise W 202 1 xvda1 10 40 0.81 |
| 80 | 1873 supervise W 202 1 xvda1 10 40 0.81 |
| 81 | 1871 supervise W 202 1 xvda1 10 40 1.03 |
| 82 | 1876 supervise W 202 1 xvda1 10 40 0.76 |
| 83 | 1892 supervise W 202 1 xvda1 10 40 0.74 |
| 84 | 1878 supervise W 202 1 xvda1 10 40 0.94 |
| 85 | 1886 supervise W 202 1 xvda1 10 40 0.76 |
| 86 | 1894 supervise W 202 1 xvda1 10 40 0.69 |
| 87 | 1869 supervise W 202 1 xvda1 10 40 0.72 |
| 88 | 1888 supervise W 202 1 xvda1 10 40 1.70 |
| 89 | 22199 bash R 202 1 xvda1 2 20 0.35 |
| 90 | 482 xfsaild/md0 W 202 16 xvdb 5 13 0.27 |
| 91 | 482 xfsaild/md0 W 202 32 xvdc 2 8 0.33 |
| 92 | 31331 pickup R 202 1 xvda1 1 4 0.31 |
| 93 | |
| 94 | 08:09:54 loadavg: 0.51 0.45 0.39 2/282 22346 |
| 95 | |
| 96 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 97 | 22069 dd R 202 1 xvda1 14689 117512 0.32 |
| 98 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.33 |
| 99 | 1880 supervise W 202 1 xvda1 10 40 0.65 |
| 100 | 1873 supervise W 202 1 xvda1 10 40 1.08 |
| 101 | 1871 supervise W 202 1 xvda1 10 40 0.66 |
| 102 | 1876 supervise W 202 1 xvda1 10 40 0.79 |
| 103 | 1892 supervise W 202 1 xvda1 10 40 0.67 |
| 104 | 1878 supervise W 202 1 xvda1 10 40 0.66 |
| 105 | 1886 supervise W 202 1 xvda1 10 40 1.02 |
| 106 | 1894 supervise W 202 1 xvda1 10 40 0.88 |
| 107 | 1869 supervise W 202 1 xvda1 10 40 0.89 |
| 108 | 1888 supervise W 202 1 xvda1 10 40 1.25 |
| 109 | |
| 110 | 08:09:59 loadavg: 0.55 0.46 0.40 2/282 22461 |
| 111 | |
| 112 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 113 | 22069 dd R 202 1 xvda1 14442 115536 0.33 |
| 114 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.46 |
| 115 | 1880 supervise W 202 1 xvda1 10 40 0.87 |
| 116 | 1873 supervise W 202 1 xvda1 10 40 0.87 |
| 117 | 1871 supervise W 202 1 xvda1 10 40 0.78 |
| 118 | 1876 supervise W 202 1 xvda1 10 40 0.86 |
| 119 | 1892 supervise W 202 1 xvda1 10 40 0.89 |
| 120 | 1878 supervise W 202 1 xvda1 10 40 0.87 |
| 121 | 1886 supervise W 202 1 xvda1 10 40 0.86 |
| 122 | 1894 supervise W 202 1 xvda1 10 40 1.06 |
| 123 | 1869 supervise W 202 1 xvda1 10 40 1.12 |
| 124 | 1888 supervise W 202 1 xvda1 10 40 0.98 |
| 125 | |
| 126 | 08:10:04 loadavg: 0.59 0.47 0.40 3/282 22576 |
| 127 | |
| 128 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 129 | 22069 dd R 202 1 xvda1 14179 113432 0.34 |
| 130 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.39 |
| 131 | 1880 supervise W 202 1 xvda1 10 40 0.81 |
| 132 | 1873 supervise W 202 1 xvda1 10 40 1.02 |
| 133 | 1871 supervise W 202 1 xvda1 10 40 1.15 |
| 134 | 1876 supervise W 202 1 xvda1 10 40 1.10 |
| 135 | 1892 supervise W 202 1 xvda1 10 40 0.77 |
| 136 | 1878 supervise W 202 1 xvda1 10 40 0.72 |
| 137 | 1886 supervise W 202 1 xvda1 10 40 0.81 |
| 138 | 1894 supervise W 202 1 xvda1 10 40 0.86 |
| 139 | 1869 supervise W 202 1 xvda1 10 40 0.83 |
| 140 | 1888 supervise W 202 1 xvda1 10 40 0.79 |
| 141 | 24467 kworker/0:2 R 202 32 xvdc 3 12 0.26 |
| 142 | 1056 cron R 202 1 xvda1 2 8 0.30 |
| 143 | 24467 kworker/0:2 R 202 16 xvdb 1 4 0.23 |
| 144 | |
| 145 | 08:10:09 loadavg: 0.54 0.46 0.40 2/281 22668 |
| 146 | |
| 147 | PID COMM D MAJ MIN DISK I/O Kbytes AVGms |
| 148 | 22069 dd R 202 1 xvda1 250 2000 0.34 |
| 149 | 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.40 |
| 150 | 1880 supervise W 202 1 xvda1 8 32 0.93 |
| 151 | 1873 supervise W 202 1 xvda1 8 32 0.76 |
| 152 | 1871 supervise W 202 1 xvda1 8 32 0.60 |
| 153 | 1876 supervise W 202 1 xvda1 8 32 0.61 |
| 154 | 1892 supervise W 202 1 xvda1 8 32 0.68 |
| 155 | 1878 supervise W 202 1 xvda1 8 32 0.90 |
| 156 | 1886 supervise W 202 1 xvda1 8 32 0.57 |
| 157 | 1894 supervise W 202 1 xvda1 8 32 0.97 |
| 158 | 1869 supervise W 202 1 xvda1 8 32 0.69 |
| 159 | 1888 supervise W 202 1 xvda1 8 32 0.67 |
| 160 | |
| 161 | This shows another "dd" command reading from xvda1. On this system, various |
| 162 | "supervise" processes do 8 disk writes per second, every second (they are |
| 163 | creating and updating "status" files). |
| 164 | |
| 165 | |
| 166 | USAGE message: |
| 167 | |
| 168 | # ./biotop.py -h |
| 169 | usage: biotop.py [-h] [-C] [-r MAXROWS] [interval] [count] |
| 170 | |
| 171 | Block device (disk) I/O by process |
| 172 | |
| 173 | positional arguments: |
| 174 | interval output interval, in seconds |
| 175 | count number of outputs |
| 176 | |
| 177 | optional arguments: |
| 178 | -h, --help show this help message and exit |
| 179 | -C, --noclear don't clear the screen |
| 180 | -r MAXROWS, --maxrows MAXROWS |
| 181 | maximum rows to print, default 20 |
| 182 | |
| 183 | examples: |
| 184 | ./biotop # block device I/O top, 1 second refresh |
| 185 | ./biotop -C # don't clear the screen |
| 186 | ./biotop 5 # 5 second summaries |
| 187 | ./biotop 5 10 # 5 second summaries, 10 times only |