1 Demonstrations of slabratetop, the Linux eBPF/bcc version. 2 3 4 slabratetop shows the rate of allocations and total bytes from the kernel 5 memory allocation caches (SLAB or SLUB), in a top-like display that refreshes. 6 For example: 7 8 # ./slabratetop 9 <screen clears> 10 07:01:35 loadavg: 0.38 0.21 0.12 1/342 13297 11 12 CACHE ALLOCS BYTES 13 kmalloc-4096 3554 14557184 14 kmalloc-256 2382 609536 15 cred_jar 2568 493056 16 anon_vma_chain 2007 128448 17 anon_vma 972 77760 18 sighand_cache 24 50688 19 mm_struct 49 50176 20 RAW 52 49920 21 proc_inode_cache 59 38232 22 signal_cache 24 26112 23 dentry 135 25920 24 sock_inode_cache 29 18560 25 files_cache 24 16896 26 inode_cache 13 7696 27 TCP 2 3840 28 pid 24 3072 29 sigqueue 17 2720 30 ext4_inode_cache 2 2160 31 buffer_head 16 1664 32 xfs_trans 5 1160 33 34 By default the screen refreshes every one second, and only the top 20 caches 35 are shown. These can be tuned with options: see USAGE (-h). 36 37 The output above showed that the kmalloc-4096 cache allocated the most, about 38 14 Mbytes during this interval. This is a generic cache; other caches have 39 more meaningful names ("dentry", "TCP", "pid", etc). 40 41 slabtop(1) is a similar tool that shows the current static volume and usage 42 of the caches. slabratetop shows the active call rates and total size of the 43 allocations. 44 45 46 Since "kmalloc-4096" isn't very descriptive, I'm interested in seeing the 47 kernel stacks that led to this allocation. In the future (maybe by now) the 48 bcc trace tool could do this. As I'm writing this, it can't, so I'll use my 49 older ftrace-based kprobe tool as a workarond. This is from my perf-tools 50 collection: https://github.com/brendangregg/perf-tools. 51 52 # ./perf-tools/bin/kprobe -s 'p:kmem_cache_alloc name=+0(+96(%di)):string' 'name == "kmalloc-4096' | head -100 53 Tracing kprobe kmem_cache_alloc. Ctrl-C to end. 54 kprobe-3892 [002] d... 7888274.478331: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" 55 kprobe-3892 [002] d... 7888274.478333: <stack trace> 56 => kmem_cache_alloc 57 => user_path_at_empty 58 => vfs_fstatat 59 => SYSC_newstat 60 => SyS_newstat 61 => entry_SYSCALL_64_fastpath 62 kprobe-3892 [002] d... 7888274.478340: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" 63 kprobe-3892 [002] d... 7888274.478341: <stack trace> 64 => kmem_cache_alloc 65 => user_path_at_empty 66 => vfs_fstatat 67 => SYSC_newstat 68 => SyS_newstat 69 => entry_SYSCALL_64_fastpath 70 kprobe-3892 [002] d... 7888274.478345: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" 71 kprobe-3892 [002] d... 7888274.478346: <stack trace> 72 => kmem_cache_alloc 73 => user_path_at_empty 74 => vfs_fstatat 75 => SYSC_newstat 76 => SyS_newstat 77 => entry_SYSCALL_64_fastpath 78 kprobe-3892 [002] d... 7888274.478350: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" 79 kprobe-3892 [002] d... 7888274.478351: <stack trace> 80 => kmem_cache_alloc 81 => user_path_at_empty 82 => vfs_fstatat 83 => SYSC_newstat 84 => SyS_newstat 85 => entry_SYSCALL_64_fastpath 86 kprobe-3892 [002] d... 7888274.478355: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" 87 kprobe-3892 [002] d... 7888274.478355: <stack trace> 88 => kmem_cache_alloc 89 => user_path_at_empty 90 => vfs_fstatat 91 => SYSC_newstat 92 => SyS_newstat 93 => entry_SYSCALL_64_fastpath 94 kprobe-3892 [002] d... 7888274.478359: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" 95 kprobe-3892 [002] d... 7888274.478359: <stack trace> 96 => kmem_cache_alloc 97 => user_path_at_empty 98 => vfs_fstatat 99 => SYSC_newstat 100 => SyS_newstat 101 => entry_SYSCALL_64_fastpath 102 [...] 103 104 This is just an example so that you can see it's possible to dig further. 105 Please don't copy-n-paste that kprobe command, as it's unlikely to work (the 106 "+0(+96(%di))" text is specific to a kernel version and architecture). 107 108 So these allocations are coming from user_path_at_empty(), which calls other 109 functions (not seen in the stack: I suspect it's a tail-call compiler 110 optimization). 111 112 113 USAGE: 114 115 # ./slabratetop -h 116 usage: slabratetop [-h] [-C] [-r MAXROWS] [interval] [count] 117 118 Kernel SLAB/SLUB memory cache allocation rate top 119 120 positional arguments: 121 interval output interval, in seconds 122 count number of outputs 123 124 optional arguments: 125 -h, --help show this help message and exit 126 -C, --noclear don't clear the screen 127 -r MAXROWS, --maxrows MAXROWS 128 maximum rows to print, default 20 129 130 examples: 131 ./slabratetop # kmem_cache_alloc() top, 1 second refresh 132 ./slabratetop -C # don't clear the screen 133 ./slabratetop 5 # 5 second summaries 134 ./slabratetop 5 10 # 5 second summaries, 10 times only 135
