Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 1 | Demonstrations of memleak. |
| 2 | |
| 3 | |
| 4 | memleak traces and matches memory allocation and deallocation requests, and |
| 5 | collects call stacks for each allocation. memleak can then print a summary |
| 6 | of which call stacks performed allocations that weren't subsequently freed. |
| 7 | For example: |
| 8 | |
| 9 | # ./memleak.py -p $(pidof allocs) |
| 10 | Attaching to malloc and free in pid 5193, Ctrl+C to quit. |
| 11 | *** Outstanding allocations: |
| 12 | 80 bytes in 5 allocations from stack |
| 13 | main+0x6d [/home/vagrant/allocs] (400862) |
| 14 | __libc_start_main+0xf0 [/usr/lib64/libc-2.21.so] (7fd460ac2790) |
| 15 | |
| 16 | *** Outstanding allocations: |
| 17 | 160 bytes in 10 allocations from stack |
| 18 | main+0x6d [/home/vagrant/allocs] (400862) |
| 19 | __libc_start_main+0xf0 [/usr/lib64/libc-2.21.so] (7fd460ac2790) |
| 20 | |
| 21 | |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame^] | 22 | As time goes on, it becomes apparent that the main function in the allocs |
| 23 | process is leaking memory, 16 bytes at a time. Fortunately, you don't have to |
| 24 | inspect each allocation individually -- you get a nice summary of which stack |
| 25 | is responsible for a large leak. |
| 26 | |
| 27 | Occasionally, you do want the individual allocation details. Perhaps the same |
| 28 | stack is allocating various sizes and you want to confirm which sizes are |
| 29 | prevalent. Use the -a switch: |
| 30 | |
| 31 | # ./memleak.py -p $(pidof allocs) -a |
| 32 | Attaching to malloc and free in pid 5193, Ctrl+C to quit. |
| 33 | *** Outstanding allocations: |
| 34 | addr = 948cd0 size = 16 |
| 35 | addr = 948d10 size = 16 |
| 36 | addr = 948d30 size = 16 |
| 37 | addr = 948cf0 size = 16 |
| 38 | 64 bytes in 4 allocations from stack |
| 39 | main+0x6d [/home/vagrant/allocs] (400862) |
| 40 | __libc_start_main+0xf0 [/usr/lib64/libc-2.21.so] (7fd460ac2790) |
| 41 | |
| 42 | *** Outstanding allocations: |
| 43 | addr = 948d50 size = 16 |
| 44 | addr = 948cd0 size = 16 |
| 45 | addr = 948d10 size = 16 |
| 46 | addr = 948d30 size = 16 |
| 47 | addr = 948cf0 size = 16 |
| 48 | addr = 948dd0 size = 16 |
| 49 | addr = 948d90 size = 16 |
| 50 | addr = 948db0 size = 16 |
| 51 | addr = 948d70 size = 16 |
| 52 | addr = 948df0 size = 16 |
| 53 | 160 bytes in 10 allocations from stack |
| 54 | main+0x6d [/home/vagrant/allocs] (400862) |
| 55 | __libc_start_main+0xf0 [/usr/lib64/libc-2.21.so] (7fd460ac2790) |
| 56 | |
| 57 | |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 58 | When using the -p switch, memleak traces the allocations of a particular |
| 59 | process. Without this switch, kernel allocations (kmalloc) are traced instead. |
| 60 | For example: |
| 61 | |
| 62 | # ./memleak.py |
| 63 | Attaching to kmalloc and kfree, Ctrl+C to quit. |
| 64 | ... |
| 65 | 248 bytes in 4 allocations from stack |
| 66 | bpf_prog_load [kernel] (ffffffff8118c471) |
| 67 | sys_bpf [kernel] (ffffffff8118c8b5) |
| 68 | |
| 69 | 328 bytes in 1 allocations from stack |
| 70 | perf_mmap [kernel] (ffffffff811990fd) |
| 71 | mmap_region [kernel] (ffffffff811df5d4) |
| 72 | do_mmap [kernel] (ffffffff811dfb83) |
| 73 | vm_mmap_pgoff [kernel] (ffffffff811c494f) |
| 74 | sys_mmap_pgoff [kernel] (ffffffff811ddf02) |
| 75 | sys_mmap [kernel] (ffffffff8101b0ab) |
| 76 | |
| 77 | 464 bytes in 1 allocations from stack |
| 78 | traceprobe_command [kernel] (ffffffff81187cf2) |
| 79 | traceprobe_probes_write [kernel] (ffffffff81187d86) |
| 80 | probes_write [kernel] (ffffffff81181580) |
| 81 | __vfs_write [kernel] (ffffffff812237b7) |
| 82 | vfs_write [kernel] (ffffffff81223ec6) |
| 83 | sys_write [kernel] (ffffffff81224b85) |
| 84 | entry_SYSCALL_64_fastpath [kernel] (ffffffff8178182e) |
| 85 | |
| 86 | 8192 bytes in 1 allocations from stack |
| 87 | alloc_and_copy_ftrace_hash.constprop.59 [kernel] (ffffffff8115d17e) |
| 88 | ftrace_set_hash [kernel] (ffffffff8115e767) |
| 89 | ftrace_set_filter_ip [kernel] (ffffffff8115e9a8) |
| 90 | arm_kprobe [kernel] (ffffffff81148600) |
| 91 | enable_kprobe [kernel] (ffffffff811486f6) |
| 92 | kprobe_register [kernel] (ffffffff81182399) |
| 93 | perf_trace_init [kernel] (ffffffff8117c4e0) |
| 94 | perf_tp_event_init [kernel] (ffffffff81192479) |
| 95 | |
| 96 | |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame^] | 97 | Here you can see that arming the kprobe to which our eBPF program is attached |
| 98 | consumed 8KB of memory. Loading the BPF program also consumed a couple hundred |
| 99 | bytes (in bpf_prog_load). |
| 100 | |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 101 | memleak stores each allocated block along with its size, timestamp, and the |
| 102 | stack that allocated it. When the block is deleted, this information is freed |
| 103 | to reduce the memory overhead. |
| 104 | |
| 105 | To avoid false positives, allocations younger than a certain age (500ms by |
| 106 | default) are not printed. To change this threshold, use the -o switch. |
| 107 | |
| 108 | By default, memleak prints its output every 5 seconds. To change this |
| 109 | interval, use the -i switch. |
| 110 | |
| 111 | |
| 112 | USAGE message: |
| 113 | |
| 114 | # ./memleak.py -h |
| 115 | usage: memleak.py [-h] [-p PID] [-t] [-i INTERVAL] [-a] [-o OLDER] |
| 116 | [-c COMMAND] |
| 117 | |
| 118 | Trace outstanding memory allocations that weren't freed. |
| 119 | Supports both user-mode allocations made with malloc/free and kernel-mode |
| 120 | allocations made with kmalloc/kfree. |
| 121 | |
| 122 | optional arguments: |
| 123 | -h, --help show this help message and exit |
| 124 | -p PID, --pid PID the PID to trace; if not specified, trace kernel |
| 125 | allocs |
| 126 | -t, --trace print trace messages for each alloc/free call |
| 127 | -i INTERVAL, --interval INTERVAL |
| 128 | interval in seconds to print outstanding allocations |
| 129 | -a, --show-allocs show allocation addresses and sizes as well as call |
| 130 | stacks |
| 131 | -o OLDER, --older OLDER |
| 132 | prune allocations younger than this age in |
| 133 | milliseconds |
| 134 | -c COMMAND, --command COMMAND |
| 135 | execute and trace the specified command |
| 136 | |
| 137 | EXAMPLES: |
| 138 | |
| 139 | ./memleak.py -p $(pidof allocs) |
| 140 | Trace allocations and display a summary of "leaked" (outstanding) |
| 141 | allocations every 5 seconds |
| 142 | ./memleak.py -p $(pidof allocs) -t |
| 143 | Trace allocations and display each individual call to malloc/free |
| 144 | ./memleak.py -p $(pidof allocs) -a -i 10 Trace allocations and display allocated addresses, sizes, and stacks every 10 seconds for outstanding allocations ./memleak.py -c "./allocs" Run the specified command and trace its allocations ./memleak.py Trace allocations in kernel mode and display a summary of outstanding allocations every 5 seconds ./memleak.py -o 60000 Trace allocations in kernel mode and display a summary of outstanding allocations that are at least one minute (60 seconds) old |
| 145 | |