Home | History | Annotate | Download | only in tools
      1 Demonstrations of memleak.
      2 
      3 
      4 memleak traces and matches memory allocation and deallocation requests, and
      5 collects call stacks for each allocation. memleak can then print a summary
      6 of which call stacks performed allocations that weren't subsequently freed.
      7 For example:
      8 
      9 # ./memleak -p $(pidof allocs)
     10 Attaching to pid 5193, Ctrl+C to quit.
     11 [11:16:33] Top 2 stacks with outstanding allocations:
     12         80 bytes in 5 allocations from stack
     13                  main+0x6d [allocs]
     14                  __libc_start_main+0xf0 [libc-2.21.so]
     15 
     16 [11:16:34] Top 2 stacks with outstanding allocations:
     17         160 bytes in 10 allocations from stack
     18                  main+0x6d [allocs]
     19                  __libc_start_main+0xf0 [libc-2.21.so]
     20 
     21 
     22 Each entry printed is a set of allocations that originate from the same call
     23 stack, and that weren't freed yet. The number of bytes and number of allocs
     24 are followed by the call stack, top to bottom, of the allocation site.
     25 
     26 As time goes on, it becomes apparent that the main function in the allocs
     27 process is leaking memory, 16 bytes at a time. Fortunately, you don't have to
     28 inspect each allocation individually -- you get a nice summary of which stack
     29 is responsible for a large leak.
     30 
     31 Occasionally, you do want the individual allocation details. Perhaps the same
     32 stack is allocating various sizes and you want to confirm which sizes are 
     33 prevalent. Use the -a switch:
     34 
     35 # ./memleak -p $(pidof allocs) -a
     36 Attaching to pid 5193, Ctrl+C to quit.
     37 [11:16:33] Top 2 stacks with outstanding allocations:
     38         addr = 948cd0 size = 16
     39         addr = 948d10 size = 16
     40         addr = 948d30 size = 16
     41         addr = 948cf0 size = 16
     42         64 bytes in 4 allocations from stack
     43                  main+0x6d [allocs]
     44                  __libc_start_main+0xf0 [libc-2.21.so]
     45 
     46 [11:16:34] Top 2 stacks with outstanding allocations:
     47         addr = 948d50 size = 16
     48         addr = 948cd0 size = 16
     49         addr = 948d10 size = 16
     50         addr = 948d30 size = 16
     51         addr = 948cf0 size = 16
     52         addr = 948dd0 size = 16
     53         addr = 948d90 size = 16
     54         addr = 948db0 size = 16
     55         addr = 948d70 size = 16
     56         addr = 948df0 size = 16
     57         160 bytes in 10 allocations from stack
     58                  main+0x6d [allocs]
     59                  __libc_start_main+0xf0 [libc-2.21.so]
     60 
     61 
     62 When using the -p switch, memleak traces the libc allocations of a particular
     63 process. Without this switch, kernel allocations are traced instead.
     64 For example:
     65 
     66 # ./memleak
     67 Attaching to kernel allocators, Ctrl+C to quit.
     68 ...
     69         248 bytes in 4 allocations from stack
     70                  bpf_prog_load [kernel]
     71                  sys_bpf [kernel]
     72 
     73         328 bytes in 1 allocations from stack
     74                  perf_mmap [kernel]
     75                  mmap_region [kernel]
     76                  do_mmap [kernel]
     77                  vm_mmap_pgoff [kernel]
     78                  sys_mmap_pgoff [kernel]
     79                  sys_mmap [kernel]
     80 
     81         464 bytes in 1 allocations from stack
     82                  traceprobe_command [kernel]
     83                  traceprobe_probes_write [kernel]
     84                  probes_write [kernel]
     85                  __vfs_write [kernel]
     86                  vfs_write [kernel]
     87                  sys_write [kernel]
     88                  entry_SYSCALL_64_fastpath [kernel]
     89 
     90         8192 bytes in 1 allocations from stack
     91                  alloc_and_copy_ftrace_hash.constprop.59 [kernel]
     92                  ftrace_set_hash [kernel]
     93                  ftrace_set_filter_ip [kernel]
     94                  arm_kprobe [kernel]
     95                  enable_kprobe [kernel]
     96                  kprobe_register [kernel]
     97                  perf_trace_init [kernel]
     98                  perf_tp_event_init [kernel]
     99 
    100 
    101 Here you can see that arming the kprobe to which our eBPF program is attached
    102 consumed 8KB of memory. Loading the BPF program also consumed a couple hundred
    103 bytes (in bpf_prog_load).
    104 
    105 memleak stores each allocated block along with its size, timestamp, and the
    106 stack that allocated it. When the block is deleted, this information is freed
    107 to reduce the memory overhead.
    108 
    109 To avoid false positives, allocations younger than a certain age (500ms by
    110 default) are not printed. To change this threshold, use the -o switch.
    111 
    112 By default, memleak prints its output every 5 seconds. To change this 
    113 interval, pass the interval as a positional parameter to memleak. You can 
    114 also control the number of times the output will be printed before exiting.
    115 For example:
    116 
    117 # ./memleak 1 10
    118 
    119 ... will print the outstanding allocation statistics every second, for ten
    120 times, and then exit. 
    121 
    122 memleak may introduce considerable overhead if your application or kernel is
    123 allocating and freeing memory at a very high rate. In that case, you can 
    124 control the overhead by sampling every N-th allocation. For example, to sample
    125 roughly 10% of the allocations and print the outstanding allocations every 5
    126 seconds, 3 times before quitting:
    127 
    128 # ./memleak -p $(pidof allocs) -s 10 5 3
    129 Attaching to pid 2614, Ctrl+C to quit.
    130 [11:16:33] Top 2 stacks with outstanding allocations:
    131         16 bytes in 1 allocations from stack
    132                  main+0x6d [allocs]
    133                  __libc_start_main+0xf0 [libc-2.21.so]
    134 
    135 [11:16:38] Top 2 stacks with outstanding allocations:
    136         16 bytes in 1 allocations from stack
    137                  main+0x6d [allocs]
    138                  __libc_start_main+0xf0 [libc-2.21.so]
    139 
    140 [11:16:43] Top 2 stacks with outstanding allocations:
    141         32 bytes in 2 allocations from stack
    142                  main+0x6d [allocs]
    143                  __libc_start_main+0xf0 [libc-2.21.so]
    144 
    145 Note that even though the application leaks 16 bytes of memory every second, 
    146 the report (printed every 5 seconds) doesn't "see" all the allocations because
    147 of the sampling rate applied. 
    148 
    149 
    150 USAGE message:
    151 
    152 # ./memleak -h
    153 usage: memleak.py [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND]
    154                   [--combined-only] [-s SAMPLE_RATE] [-T TOP] [-z MIN_SIZE]
    155                   [-Z MAX_SIZE] [-O OBJ]
    156                   [interval] [count]
    157 
    158 Trace outstanding memory allocations that weren't freed.
    159 Supports both user-mode allocations made with libc functions and kernel-mode
    160 allocations made with kmalloc/kmem_cache_alloc/get_free_pages and corresponding
    161 memory release functions.
    162 
    163 positional arguments:
    164   interval              interval in seconds to print outstanding allocations
    165   count                 number of times to print the report before exiting
    166 
    167 optional arguments:
    168   -h, --help            show this help message and exit
    169   -p PID, --pid PID     the PID to trace; if not specified, trace kernel
    170                         allocs
    171   -t, --trace           print trace messages for each alloc/free call
    172   -a, --show-allocs     show allocation addresses and sizes as well as call
    173                         stacks
    174   -o OLDER, --older OLDER
    175                         prune allocations younger than this age in
    176                         milliseconds
    177   -c COMMAND, --command COMMAND
    178                         execute and trace the specified command
    179   --combined-only       show combined allocation statistics only
    180   -s SAMPLE_RATE, --sample-rate SAMPLE_RATE
    181                         sample every N-th allocation to decrease the overhead
    182   -T TOP, --top TOP     display only this many top allocating stacks (by size)
    183   -z MIN_SIZE, --min-size MIN_SIZE
    184                         capture only allocations larger than this size
    185   -Z MAX_SIZE, --max-size MAX_SIZE
    186                         capture only allocations smaller than this size
    187   -O OBJ, --obj OBJ     attach to allocator functions in the specified object
    188 
    189 EXAMPLES:
    190 
    191 ./memleak -p $(pidof allocs)
    192         Trace allocations and display a summary of "leaked" (outstanding)
    193         allocations every 5 seconds
    194 ./memleak -p $(pidof allocs) -t
    195         Trace allocations and display each individual allocator function call
    196 ./memleak -ap $(pidof allocs) 10
    197         Trace allocations and display allocated addresses, sizes, and stacks
    198         every 10 seconds for outstanding allocations
    199 ./memleak -c "./allocs"
    200         Run the specified command and trace its allocations
    201 ./memleak
    202         Trace allocations in kernel mode and display a summary of outstanding
    203         allocations every 5 seconds
    204 ./memleak -o 60000
    205         Trace allocations in kernel mode and display a summary of outstanding
    206         allocations that are at least one minute (60 seconds) old
    207 ./memleak -s 5
    208         Trace roughly every 5th allocation, to reduce overhead
    209