You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
134 lines
5.2 KiB
134 lines
5.2 KiB
Demonstrations of slabratetop, the Linux eBPF/bcc version. |
|
|
|
|
|
slabratetop shows the rate of allocations and total bytes from the kernel |
|
memory allocation caches (SLAB or SLUB), in a top-like display that refreshes. |
|
For example: |
|
|
|
# ./slabratetop |
|
<screen clears> |
|
07:01:35 loadavg: 0.38 0.21 0.12 1/342 13297 |
|
|
|
CACHE ALLOCS BYTES |
|
kmalloc-4096 3554 14557184 |
|
kmalloc-256 2382 609536 |
|
cred_jar 2568 493056 |
|
anon_vma_chain 2007 128448 |
|
anon_vma 972 77760 |
|
sighand_cache 24 50688 |
|
mm_struct 49 50176 |
|
RAW 52 49920 |
|
proc_inode_cache 59 38232 |
|
signal_cache 24 26112 |
|
dentry 135 25920 |
|
sock_inode_cache 29 18560 |
|
files_cache 24 16896 |
|
inode_cache 13 7696 |
|
TCP 2 3840 |
|
pid 24 3072 |
|
sigqueue 17 2720 |
|
ext4_inode_cache 2 2160 |
|
buffer_head 16 1664 |
|
xfs_trans 5 1160 |
|
|
|
By default the screen refreshes every one second, and only the top 20 caches |
|
are shown. These can be tuned with options: see USAGE (-h). |
|
|
|
The output above showed that the kmalloc-4096 cache allocated the most, about |
|
14 Mbytes during this interval. This is a generic cache; other caches have |
|
more meaningful names ("dentry", "TCP", "pid", etc). |
|
|
|
slabtop(1) is a similar tool that shows the current static volume and usage |
|
of the caches. slabratetop shows the active call rates and total size of the |
|
allocations. |
|
|
|
|
|
Since "kmalloc-4096" isn't very descriptive, I'm interested in seeing the |
|
kernel stacks that led to this allocation. In the future (maybe by now) the |
|
bcc trace tool could do this. As I'm writing this, it can't, so I'll use my |
|
older ftrace-based kprobe tool as a workarond. This is from my perf-tools |
|
collection: https://github.com/brendangregg/perf-tools. |
|
|
|
# ./perf-tools/bin/kprobe -s 'p:kmem_cache_alloc name=+0(+96(%di)):string' 'name == "kmalloc-4096' | head -100 |
|
Tracing kprobe kmem_cache_alloc. Ctrl-C to end. |
|
kprobe-3892 [002] d... 7888274.478331: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" |
|
kprobe-3892 [002] d... 7888274.478333: <stack trace> |
|
=> kmem_cache_alloc |
|
=> user_path_at_empty |
|
=> vfs_fstatat |
|
=> SYSC_newstat |
|
=> SyS_newstat |
|
=> entry_SYSCALL_64_fastpath |
|
kprobe-3892 [002] d... 7888274.478340: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" |
|
kprobe-3892 [002] d... 7888274.478341: <stack trace> |
|
=> kmem_cache_alloc |
|
=> user_path_at_empty |
|
=> vfs_fstatat |
|
=> SYSC_newstat |
|
=> SyS_newstat |
|
=> entry_SYSCALL_64_fastpath |
|
kprobe-3892 [002] d... 7888274.478345: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" |
|
kprobe-3892 [002] d... 7888274.478346: <stack trace> |
|
=> kmem_cache_alloc |
|
=> user_path_at_empty |
|
=> vfs_fstatat |
|
=> SYSC_newstat |
|
=> SyS_newstat |
|
=> entry_SYSCALL_64_fastpath |
|
kprobe-3892 [002] d... 7888274.478350: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" |
|
kprobe-3892 [002] d... 7888274.478351: <stack trace> |
|
=> kmem_cache_alloc |
|
=> user_path_at_empty |
|
=> vfs_fstatat |
|
=> SYSC_newstat |
|
=> SyS_newstat |
|
=> entry_SYSCALL_64_fastpath |
|
kprobe-3892 [002] d... 7888274.478355: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" |
|
kprobe-3892 [002] d... 7888274.478355: <stack trace> |
|
=> kmem_cache_alloc |
|
=> user_path_at_empty |
|
=> vfs_fstatat |
|
=> SYSC_newstat |
|
=> SyS_newstat |
|
=> entry_SYSCALL_64_fastpath |
|
kprobe-3892 [002] d... 7888274.478359: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" |
|
kprobe-3892 [002] d... 7888274.478359: <stack trace> |
|
=> kmem_cache_alloc |
|
=> user_path_at_empty |
|
=> vfs_fstatat |
|
=> SYSC_newstat |
|
=> SyS_newstat |
|
=> entry_SYSCALL_64_fastpath |
|
[...] |
|
|
|
This is just an example so that you can see it's possible to dig further. |
|
Please don't copy-n-paste that kprobe command, as it's unlikely to work (the |
|
"+0(+96(%di))" text is specific to a kernel version and architecture). |
|
|
|
So these allocations are coming from user_path_at_empty(), which calls other |
|
functions (not seen in the stack: I suspect it's a tail-call compiler |
|
optimization). |
|
|
|
|
|
USAGE: |
|
|
|
# ./slabratetop -h |
|
usage: slabratetop [-h] [-C] [-r MAXROWS] [interval] [count] |
|
|
|
Kernel SLAB/SLUB memory cache allocation rate top |
|
|
|
positional arguments: |
|
interval output interval, in seconds |
|
count number of outputs |
|
|
|
optional arguments: |
|
-h, --help show this help message and exit |
|
-C, --noclear don't clear the screen |
|
-r MAXROWS, --maxrows MAXROWS |
|
maximum rows to print, default 20 |
|
|
|
examples: |
|
./slabratetop # kmem_cache_alloc() top, 1 second refresh |
|
./slabratetop -C # don't clear the screen |
|
./slabratetop 5 # 5 second summaries |
|
./slabratetop 5 10 # 5 second summaries, 10 times only
|
|
|