Demonstrations of opensnoop, the Linux eBPF/bcc version.


opensnoop traces the open() syscall system-wide, and prints various details.
Example output:

# ./opensnoop
PID    COMM      FD ERR PATH
17326  <...>      7   0 /sys/kernel/debug/tracing/trace_pipe
1576   snmpd      9   0 /proc/net/dev
1576   snmpd     11   0 /proc/net/if_inet6
1576   snmpd     11   0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms
1576   snmpd     11   0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms
1576   snmpd     11   0 /proc/sys/net/ipv6/conf/eth0/forwarding
1576   snmpd     11   0 /proc/sys/net/ipv6/neigh/eth0/base_reachable_time_ms
1576   snmpd     11   0 /proc/sys/net/ipv4/neigh/lo/retrans_time_ms
1576   snmpd     11   0 /proc/sys/net/ipv6/neigh/lo/retrans_time_ms
1576   snmpd     11   0 /proc/sys/net/ipv6/conf/lo/forwarding
1576   snmpd     11   0 /proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms
1576   snmpd      9   0 /proc/diskstats
1576   snmpd      9   0 /proc/stat
1576   snmpd      9   0 /proc/vmstat
1956   supervise  9   0 supervise/status.new
1956   supervise  9   0 supervise/status.new
17358  run        3   0 /etc/ld.so.cache
17358  run        3   0 /lib/x86_64-linux-gnu/libtinfo.so.5
17358  run        3   0 /lib/x86_64-linux-gnu/libdl.so.2
17358  run        3   0 /lib/x86_64-linux-gnu/libc.so.6
17358  run       -1   6 /dev/tty
17358  run        3   0 /proc/meminfo
17358  run        3   0 /etc/nsswitch.conf
17358  run        3   0 /etc/ld.so.cache
17358  run        3   0 /lib/x86_64-linux-gnu/libnss_compat.so.2
17358  run        3   0 /lib/x86_64-linux-gnu/libnsl.so.1
17358  run        3   0 /etc/ld.so.cache
17358  run        3   0 /lib/x86_64-linux-gnu/libnss_nis.so.2
17358  run        3   0 /lib/x86_64-linux-gnu/libnss_files.so.2
17358  run        3   0 /etc/passwd
17358  run        3   0 ./run
^C

While tracing, the snmpd process opened various /proc files (reading metrics),
and a "run" process read various libraries and config files (looks like it
was starting up: a new process).

opensnoop can be useful for discovering configuration and log files, if used
during application startup.


The -p option can be used to filter on a PID, which is filtered in-kernel. Here
I've used it with -T to print timestamps:

 ./opensnoop -Tp 1956
TIME(s)       PID    COMM               FD ERR PATH
0.000000000   1956   supervise           9   0 supervise/status.new
0.000289999   1956   supervise           9   0 supervise/status.new
1.023068000   1956   supervise           9   0 supervise/status.new
1.023381997   1956   supervise           9   0 supervise/status.new
2.046030000   1956   supervise           9   0 supervise/status.new
2.046363000   1956   supervise           9   0 supervise/status.new
3.068203997   1956   supervise           9   0 supervise/status.new
3.068544999   1956   supervise           9   0 supervise/status.new

This shows the supervise process is opening the status.new file twice every
second.


The -U option include UID on output:

# ./opensnoop -U
UID   PID    COMM               FD ERR PATH
0     27063  vminfo              5   0 /var/run/utmp
103   628    dbus-daemon        -1   2 /usr/local/share/dbus-1/system-services
103   628    dbus-daemon        18   0 /usr/share/dbus-1/system-services
103   628    dbus-daemon        -1   2 /lib/dbus-1/system-services


The -u option filtering UID:

# ./opensnoop -Uu 1000
UID   PID    COMM               FD ERR PATH
1000  30240  ls                  3   0 /etc/ld.so.cache
1000  30240  ls                  3   0 /lib/x86_64-linux-gnu/libselinux.so.1
1000  30240  ls                  3   0 /lib/x86_64-linux-gnu/libc.so.6
1000  30240  ls                  3   0 /lib/x86_64-linux-gnu/libpcre.so.3
1000  30240  ls                  3   0 /lib/x86_64-linux-gnu/libdl.so.2
1000  30240  ls                  3   0 /lib/x86_64-linux-gnu/libpthread.so.0

The -x option only prints failed opens:

# ./opensnoop -x
PID    COMM      FD ERR PATH
18372  run       -1   6 /dev/tty
18373  run       -1   6 /dev/tty
18373  multilog  -1  13 lock
18372  multilog  -1  13 lock
18384  df        -1   2 /usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo
18384  df        -1   2 /usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo
18384  df        -1   2 /usr/share/locale/en_US/LC_MESSAGES/coreutils.mo
18384  df        -1   2 /usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo
18384  df        -1   2 /usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo
18384  df        -1   2 /usr/share/locale/en/LC_MESSAGES/coreutils.mo
18385  run       -1   6 /dev/tty
18386  run       -1   6 /dev/tty

This caught a df command failing to open a coreutils.mo file, and trying from
different directories.

The ERR column is the system error number. Error number 2 is ENOENT: no such
file or directory.


A maximum tracing duration can be set with the -d option. For example, to trace
for 2 seconds:

# ./opensnoop -d 2
PID    COMM               FD ERR PATH
2191   indicator-multi    11   0 /sys/block
2191   indicator-multi    11   0 /sys/block
2191   indicator-multi    11   0 /sys/block
2191   indicator-multi    11   0 /sys/block
2191   indicator-multi    11   0 /sys/block


The -n option can be used to filter on process name using partial matches:

# ./opensnoop -n ed

PID    COMM               FD ERR PATH
2679   sed                 3   0 /etc/ld.so.cache
2679   sed                 3   0 /lib/x86_64-linux-gnu/libselinux.so.1
2679   sed                 3   0 /lib/x86_64-linux-gnu/libc.so.6
2679   sed                 3   0 /lib/x86_64-linux-gnu/libpcre.so.3
2679   sed                 3   0 /lib/x86_64-linux-gnu/libdl.so.2
2679   sed                 3   0 /lib/x86_64-linux-gnu/libpthread.so.0
2679   sed                 3   0 /proc/filesystems
2679   sed                 3   0 /usr/lib/locale/locale-archive
2679   sed                -1   2
2679   sed                 3   0 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
2679   sed                 3   0 /dev/null
2680   sed                 3   0 /etc/ld.so.cache
2680   sed                 3   0 /lib/x86_64-linux-gnu/libselinux.so.1
2680   sed                 3   0 /lib/x86_64-linux-gnu/libc.so.6
2680   sed                 3   0 /lib/x86_64-linux-gnu/libpcre.so.3
2680   sed                 3   0 /lib/x86_64-linux-gnu/libdl.so.2
2680   sed                 3   0 /lib/x86_64-linux-gnu/libpthread.so.0
2680   sed                 3   0 /proc/filesystems
2680   sed                 3   0 /usr/lib/locale/locale-archive
2680   sed                -1   2
^C

This caught the 'sed' command because it partially matches 'ed' that's passed
to the '-n' option.


The -e option prints out extra columns; for example, the following output
contains the flags passed to open(2), in octal:

# ./opensnoop -e
PID    COMM               FD ERR FLAGS    PATH
28512  sshd               10   0 00101101 /proc/self/oom_score_adj
28512  sshd                3   0 02100000 /etc/ld.so.cache
28512  sshd                3   0 02100000 /lib/x86_64-linux-gnu/libwrap.so.0
28512  sshd                3   0 02100000 /lib/x86_64-linux-gnu/libaudit.so.1
28512  sshd                3   0 02100000 /lib/x86_64-linux-gnu/libpam.so.0
28512  sshd                3   0 02100000 /lib/x86_64-linux-gnu/libselinux.so.1
28512  sshd                3   0 02100000 /lib/x86_64-linux-gnu/libsystemd.so.0
28512  sshd                3   0 02100000 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
28512  sshd                3   0 02100000 /lib/x86_64-linux-gnu/libutil.so.1


The -f option filters based on flags to the open(2) call, for example:

# ./opensnoop -e -f O_WRONLY -f O_RDWR
PID    COMM               FD ERR FLAGS    PATH
28084  clear_console       3   0 00100002 /dev/tty
28084  clear_console      -1  13 00100002 /dev/tty0
28084  clear_console      -1  13 00100001 /dev/tty0
28084  clear_console      -1  13 00100002 /dev/console
28084  clear_console      -1  13 00100001 /dev/console
28051  sshd                8   0 02100002 /var/run/utmp
28051  sshd                7   0 00100001 /var/log/wtmp


USAGE message:

# ./opensnoop -h
usage: opensnoop [-h] [-T] [-x] [-p PID] [-t TID] [-d DURATION] [-n NAME]
                 [-e] [-f FLAG_FILTER]

Trace open() syscalls

optional arguments:
  -h, --help            show this help message and exit
  -T, --timestamp       include timestamp on output
  -U, --print-uid       include UID on output
  -x, --failed          only show failed opens
  -p PID, --pid PID     trace this PID only
  -t TID, --tid TID     trace this TID only
  -u UID, --uid UID     trace this UID only
  -d DURATION, --duration DURATION
                        total duration of trace in seconds
  -n NAME, --name NAME  only print process names containing this name
  -e, --extended_fields
                        show extended fields
  -f FLAG_FILTER, --flag_filter FLAG_FILTER
                        filter on flags argument (e.g., O_WRONLY)

examples:
    ./opensnoop           # trace all open() syscalls
    ./opensnoop -T        # include timestamps
    ./opensnoop -U        # include UID
    ./opensnoop -x        # only show failed opens
    ./opensnoop -p 181    # only trace PID 181
    ./opensnoop -t 123    # only trace TID 123
    ./opensnoop -u 1000   # only trace UID 1000
    ./opensnoop -d 10     # trace for 10 seconds only
    ./opensnoop -n main   # only print process names containing "main"
    ./opensnoop -e        # show extended fields
    ./opensnoop -f O_WRONLY -f O_RDWR  # only print calls for writing