Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 1 | Demonstrations of memleak. |
| 2 | |
| 3 | |
| 4 | memleak traces and matches memory allocation and deallocation requests, and |
| 5 | collects call stacks for each allocation. memleak can then print a summary |
| 6 | of which call stacks performed allocations that weren't subsequently freed. |
| 7 | For example: |
| 8 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 9 | # ./memleak -p $(pidof allocs) |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 10 | Attaching to malloc and free in pid 5193, Ctrl+C to quit. |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 11 | [11:16:33] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 12 | 80 bytes in 5 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 13 | main+0x6d [allocs] |
| 14 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 15 | |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 16 | [11:16:34] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 17 | 160 bytes in 10 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 18 | main+0x6d [allocs] |
| 19 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 20 | |
| 21 | |
Sasha Goldshtein | b9b9ad3 | 2016-02-08 03:41:43 -0800 | [diff] [blame] | 22 | Each entry printed is a set of allocations that originate from the same call |
| 23 | stack, and that weren't freed yet. The number of bytes and number of allocs |
| 24 | are followed by the call stack, top to bottom, of the allocation site. |
| 25 | |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 26 | As time goes on, it becomes apparent that the main function in the allocs |
| 27 | process is leaking memory, 16 bytes at a time. Fortunately, you don't have to |
| 28 | inspect each allocation individually -- you get a nice summary of which stack |
| 29 | is responsible for a large leak. |
| 30 | |
| 31 | Occasionally, you do want the individual allocation details. Perhaps the same |
| 32 | stack is allocating various sizes and you want to confirm which sizes are |
| 33 | prevalent. Use the -a switch: |
| 34 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 35 | # ./memleak -p $(pidof allocs) -a |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 36 | Attaching to malloc and free in pid 5193, Ctrl+C to quit. |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 37 | [11:16:33] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 38 | addr = 948cd0 size = 16 |
| 39 | addr = 948d10 size = 16 |
| 40 | addr = 948d30 size = 16 |
| 41 | addr = 948cf0 size = 16 |
| 42 | 64 bytes in 4 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 43 | main+0x6d [allocs] |
| 44 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 45 | |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 46 | [11:16:34] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 47 | addr = 948d50 size = 16 |
| 48 | addr = 948cd0 size = 16 |
| 49 | addr = 948d10 size = 16 |
| 50 | addr = 948d30 size = 16 |
| 51 | addr = 948cf0 size = 16 |
| 52 | addr = 948dd0 size = 16 |
| 53 | addr = 948d90 size = 16 |
| 54 | addr = 948db0 size = 16 |
| 55 | addr = 948d70 size = 16 |
| 56 | addr = 948df0 size = 16 |
| 57 | 160 bytes in 10 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 58 | main+0x6d [allocs] |
| 59 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 60 | |
| 61 | |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 62 | When using the -p switch, memleak traces the allocations of a particular |
| 63 | process. Without this switch, kernel allocations (kmalloc) are traced instead. |
| 64 | For example: |
| 65 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 66 | # ./memleak |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 67 | Attaching to kmalloc and kfree, Ctrl+C to quit. |
| 68 | ... |
| 69 | 248 bytes in 4 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 70 | bpf_prog_load [kernel] |
| 71 | sys_bpf [kernel] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 72 | |
| 73 | 328 bytes in 1 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 74 | perf_mmap [kernel] |
| 75 | mmap_region [kernel] |
| 76 | do_mmap [kernel] |
| 77 | vm_mmap_pgoff [kernel] |
| 78 | sys_mmap_pgoff [kernel] |
| 79 | sys_mmap [kernel] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 80 | |
| 81 | 464 bytes in 1 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 82 | traceprobe_command [kernel] |
| 83 | traceprobe_probes_write [kernel] |
| 84 | probes_write [kernel] |
| 85 | __vfs_write [kernel] |
| 86 | vfs_write [kernel] |
| 87 | sys_write [kernel] |
| 88 | entry_SYSCALL_64_fastpath [kernel] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 89 | |
| 90 | 8192 bytes in 1 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 91 | alloc_and_copy_ftrace_hash.constprop.59 [kernel] |
| 92 | ftrace_set_hash [kernel] |
| 93 | ftrace_set_filter_ip [kernel] |
| 94 | arm_kprobe [kernel] |
| 95 | enable_kprobe [kernel] |
| 96 | kprobe_register [kernel] |
| 97 | perf_trace_init [kernel] |
| 98 | perf_tp_event_init [kernel] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 99 | |
| 100 | |
Sasha Goldshtein | 33522d7 | 2016-02-08 03:39:44 -0800 | [diff] [blame] | 101 | Here you can see that arming the kprobe to which our eBPF program is attached |
| 102 | consumed 8KB of memory. Loading the BPF program also consumed a couple hundred |
| 103 | bytes (in bpf_prog_load). |
| 104 | |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 105 | memleak stores each allocated block along with its size, timestamp, and the |
| 106 | stack that allocated it. When the block is deleted, this information is freed |
| 107 | to reduce the memory overhead. |
| 108 | |
| 109 | To avoid false positives, allocations younger than a certain age (500ms by |
| 110 | default) are not printed. To change this threshold, use the -o switch. |
| 111 | |
| 112 | By default, memleak prints its output every 5 seconds. To change this |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 113 | interval, pass the interval as a positional parameter to memleak. You can |
| 114 | also control the number of times the output will be printed before exiting. |
| 115 | For example: |
| 116 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 117 | # ./memleak 1 10 |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 118 | |
| 119 | ... will print the outstanding allocation statistics every second, for ten |
| 120 | times, and then exit. |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 121 | |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 122 | memleak may introduce considerable overhead if your application or kernel is |
| 123 | allocating and freeing memory at a very high rate. In that case, you can |
| 124 | control the overhead by sampling every N-th allocation. For example, to sample |
| 125 | roughly 10% of the allocations and print the outstanding allocations every 5 |
| 126 | seconds, 3 times before quitting: |
| 127 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 128 | # ./memleak -p $(pidof allocs) -s 10 5 3 |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 129 | Attaching to malloc and free in pid 2614, Ctrl+C to quit. |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 130 | [11:16:33] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 131 | 16 bytes in 1 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 132 | main+0x6d [allocs] |
| 133 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 134 | |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 135 | [11:16:38] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 136 | 16 bytes in 1 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 137 | main+0x6d [allocs] |
| 138 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 139 | |
Sasha Goldshtein | 828edb5 | 2016-02-09 11:18:32 -0800 | [diff] [blame] | 140 | [11:16:43] Top 2 stacks with outstanding allocations: |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 141 | 32 bytes in 2 allocations from stack |
Sasha Goldshtein | 49df994 | 2017-02-08 23:22:06 -0500 | [diff] [blame] | 142 | main+0x6d [allocs] |
| 143 | __libc_start_main+0xf0 [libc-2.21.so] |
Sasha Goldshtein | d2241f4 | 2016-02-09 06:23:10 -0800 | [diff] [blame] | 144 | |
| 145 | Note that even though the application leaks 16 bytes of memory every second, |
| 146 | the report (printed every 5 seconds) doesn't "see" all the allocations because |
| 147 | of the sampling rate applied. |
| 148 | |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 149 | |
| 150 | USAGE message: |
| 151 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 152 | # ./memleak -h |
Maria Kacik | 9389ab4 | 2017-01-18 21:43:41 -0800 | [diff] [blame] | 153 | usage: memleak.py [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND] |
| 154 | [-s SAMPLE_RATE] [-T TOP] [-z MIN_SIZE] [-Z MAX_SIZE] |
| 155 | [-O OBJ] |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 156 | [interval] [count] |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 157 | |
| 158 | Trace outstanding memory allocations that weren't freed. |
| 159 | Supports both user-mode allocations made with malloc/free and kernel-mode |
| 160 | allocations made with kmalloc/kfree. |
| 161 | |
Maria Kacik | 9389ab4 | 2017-01-18 21:43:41 -0800 | [diff] [blame] | 162 | positional arguments: |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 163 | interval interval in seconds to print outstanding allocations |
| 164 | count number of times to print the report before exiting |
| 165 | |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 166 | optional arguments: |
| 167 | -h, --help show this help message and exit |
| 168 | -p PID, --pid PID the PID to trace; if not specified, trace kernel |
| 169 | allocs |
| 170 | -t, --trace print trace messages for each alloc/free call |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 171 | -a, --show-allocs show allocation addresses and sizes as well as call |
| 172 | stacks |
| 173 | -o OLDER, --older OLDER |
| 174 | prune allocations younger than this age in |
| 175 | milliseconds |
| 176 | -c COMMAND, --command COMMAND |
| 177 | execute and trace the specified command |
Sasha Goldshtein | 521ab4f | 2016-02-08 05:48:31 -0800 | [diff] [blame] | 178 | -s SAMPLE_RATE, --sample-rate SAMPLE_RATE |
| 179 | sample every N-th allocation to decrease the overhead |
Sasha Goldshtein | c8148c8 | 2016-02-09 11:15:41 -0800 | [diff] [blame] | 180 | -T TOP, --top TOP display only this many top allocating stacks (by size) |
Sasha Goldshtein | 5045964 | 2016-02-10 08:35:20 -0800 | [diff] [blame] | 181 | -z MIN_SIZE, --min-size MIN_SIZE |
| 182 | capture only allocations larger than this size |
| 183 | -Z MAX_SIZE, --max-size MAX_SIZE |
| 184 | capture only allocations smaller than this size |
Maria Kacik | 9389ab4 | 2017-01-18 21:43:41 -0800 | [diff] [blame] | 185 | -O OBJ, --obj OBJ attach to malloc & free in the specified object |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 186 | |
| 187 | EXAMPLES: |
| 188 | |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 189 | ./memleak -p $(pidof allocs) |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 190 | Trace allocations and display a summary of "leaked" (outstanding) |
| 191 | allocations every 5 seconds |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 192 | ./memleak -p $(pidof allocs) -t |
Sasha Goldshtein | dda4769 | 2016-02-08 03:10:13 -0800 | [diff] [blame] | 193 | Trace allocations and display each individual call to malloc/free |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 194 | ./memleak -ap $(pidof allocs) 10 |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 195 | Trace allocations and display allocated addresses, sizes, and stacks |
| 196 | every 10 seconds for outstanding allocations |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 197 | ./memleak -c "./allocs" |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 198 | Run the specified command and trace its allocations |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 199 | ./memleak |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 200 | Trace allocations in kernel mode and display a summary of outstanding |
| 201 | allocations every 5 seconds |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 202 | ./memleak -o 60000 |
Sasha Goldshtein | 75ba13f | 2016-02-09 06:03:46 -0800 | [diff] [blame] | 203 | Trace allocations in kernel mode and display a summary of outstanding |
| 204 | allocations that are at least one minute (60 seconds) old |
Sasha Goldshtein | 29e37d9 | 2016-02-14 06:56:07 -0800 | [diff] [blame] | 205 | ./memleak -s 5 |
Sasha Goldshtein | 521ab4f | 2016-02-08 05:48:31 -0800 | [diff] [blame] | 206 | Trace roughly every 5th allocation, to reduce overhead |