tools/filetop_example.txt

Demonstrations of filetop, the Linux eBPF/bcc version.


filetop shows reads and writes by file, with process details. For example:

# ./filetop -C
Tracing... Output every 1 secs. Hit Ctrl-C to end

08:00:23 loadavg: 0.91 0.33 0.23 3/286 26635

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
26628  ld               161    186    643     152     R built-in.o
26634  cc1              1      0      200     0       R autoconf.h
26618  cc1              1      0      200     0       R autoconf.h
26634  cc1              12     0      192     0       R tracepoint.h
26584  cc1              2      0      143     0       R mm.h
26634  cc1              2      0      143     0       R mm.h
26631  make             34     0      136     0       R auto.conf
26634  cc1              1      0      98      0       R fs.h
26584  cc1              1      0      98      0       R fs.h
26634  cc1              1      0      91      0       R sched.h
26634  cc1              1      0      78      0       R printk.c
26634  cc1              3      0      73      0       R mmzone.h
26628  ld               18     0      72      0       R hibernate.o
26628  ld               16     0      64      0       R suspend.o
26628  ld               16     0      64      0       R snapshot.o
26628  ld               16     0      64      0       R qos.o
26628  ld               13     0      52      0       R main.o
26628  ld               12     0      52      0       R swap.o
[...]

This shows various files read and written during a Linux kernel build. By
default the output is sorted by the total read size in Kbytes (R_Kb). Sorting
order can be changed via -s option. This is instrumenting at the VFS interface,
so this is reads and writes that may return entirely from the file system cache
(page cache).

While not printed, the average read and write size can be calculated by
dividing R_Kb by READS, and the same for writes.

The "T" column indicates the type of the file: "R" for regular files, "S" for
sockets, and "O" for other (including pipes). By default only regular files are
shown; use the -a option to show all file types.

This script works by tracing the vfs_read() and vfs_write() functions using
kernel dynamic tracing, which instruments explicit read and write calls. If
files are read or written using another means (eg, via mmap()), then they
will not be visible using this tool.

This should be useful for file system workload characterization when analyzing
the performance of applications.

Note that tracing VFS level reads and writes can be a frequent activity, and
this tool can begin to cost measurable overhead at high I/O rates.


A -C option will stop clearing the screen, and -r with a number will restrict
the output to that many rows (20 by default). For example, not clearing
the screen and showing the top 5 only:

# ./filetop -Cr 5
Tracing... Output every 1 secs. Hit Ctrl-C to end

08:05:11 loadavg: 0.75 0.35 0.25 3/285 822

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
32672  cksum            5006   0      320384  0       R data1
809    run              2      0      8       0       R nsswitch.conf
811    run              2      0      8       0       R nsswitch.conf
804    chown            2      0      8       0       R nsswitch.conf

08:05:12 loadavg: 0.75 0.35 0.25 3/285 845

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
32672  cksum            4986   0      319104  0       R data1
845    chown            2      0      8       0       R nsswitch.conf
828    run              2      0      8       0       R nsswitch.conf
835    run              2      0      8       0       R nsswitch.conf
830    run              2      0      8       0       R nsswitch.conf

08:05:13 loadavg: 0.75 0.35 0.25 3/285 868

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
32672  cksum            4985   0      319040  0       R data1
857    run              2      0      8       0       R nsswitch.conf
858    run              2      0      8       0       R nsswitch.conf
859    run              2      0      8       0       R nsswitch.conf
848    run              2      0      8       0       R nsswitch.conf
[...]

This output shows a cksum command reading data1.


An optional interval and optional count can also be added to the end of the
command line. For example, for 1 second interval, and 3 summaries in total:

# ./filetop -Cr 5 -a 1 3
Tracing... Output every 1 secs. Hit Ctrl-C to end

08:08:20 loadavg: 0.30 0.42 0.31 3/282 5187

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
12421  sshd             14101  0      225616  0       O ptmx
12296  sshd             4      0      64      0       O ptmx
12421  sshd             3      14104  48      778     S TCP
5178   run              2      0      8       0       R nsswitch.conf
5165   run              2      0      8       0       R nsswitch.conf

08:08:21 loadavg: 0.30 0.42 0.31 5/282 5210

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
12421  sshd             9159   0      146544  0       O ptmx
12421  sshd             3      9161   48      534     S TCP
12296  sshd             1      0      16      0       S TCP
5188   run              2      0      8       0       R nsswitch.conf
5203   run              2      0      8       0       R nsswitch.conf

08:08:22 loadavg: 0.30 0.42 0.31 2/282 5233

PID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
12421  sshd             26166  0      418656  0       O ptmx
12421  sshd             4      26171  64      1385    S TCP
12296  sshd             1      0      16      0       O ptmx
5214   run              2      0      8       0       R nsswitch.conf
5227   run              2      0      8       0       R nsswitch.conf
Detaching...

This example shows the -a option to include all file types. It caught heavy
socket I/O from an sshd process, showing up as non-regular file types (the "O"
for other, and "S" for socket, in the type column: "T").


USAGE message:

# ./filetop -h
usage: filetop.py [-h] [-a] [-C] [-r MAXROWS] [-p PID] [interval] [count]

File reads and writes by process

positional arguments:
  interval              output interval, in seconds
  count                 number of outputs

optional arguments:
  -h, --help            show this help message and exit
  -a, --all-files       include non-regular file types (sockets, FIFOs, etc)
  -C, --noclear         don't clear the screen
  -r MAXROWS, --maxrows MAXROWS
                        maximum rows to print, default 20
  -s {reads,writes,rbytes,wbytes}, --sort {reads,writes,rbytes,wbytes}
                        sort column, default rbytes
  -p PID, --pid PID     trace this PID only

examples:
    ./filetop            # file I/O top, 1 second refresh
    ./filetop -C         # don't clear the screen
    ./filetop -p 181     # PID 181 only
    ./filetop 5          # 5 second summaries
    ./filetop 5 10       # 5 second summaries, 10 times only