stackcount - Count function calls and their stack traces. Uses Linux eBPF/bcc. stackcount [-h] [-p PID] [-c CPU] [-i INTERVAL] [-D DURATION] [-T] [-r] [-s] [-P] [-K] [-U] [-v] [-d] [-f] [--debug] pattern stackcount traces functions and frequency counts them with their entire stack trace, kernel stack and user stack, summarized in-kernel for efficiency. This allows higher frequency events to be studied. The output consists of unique stack traces, and their occurrence counts. In addition to kernel and user functions, kernel tracepoints and USDT tracepoint are also supported. The pattern is a string with optional '*' wildcards, similar to file globbing. If you'd prefer to use regular expressions, use the -r option. This tool only works on Linux 4.6+. Stack traces are obtained using the new `BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old. CONFIG_BPF and bcc.

-h Print usage message.

-r Allow regular expressions for the search pattern. The default allows "*" wildcards only.

-s Show address offsets.

-P Display stacks separately for each process.

-K Show kernel stack only.

-U Show user stack only.

-T Include a timestamp with interval output.

-v Show raw addresses.

-d Print a delimiter ("--") in-between the kernel and user stacks.

--debug Print the source of the BPF program when loading it (for debugging purposes).

-i interval Summary interval, in seconds.

-D duration Total duration of trace, in seconds. -f Folded output format.

-p PID Trace this process ID only (filtered in-kernel).

-c CPU Trace this CPU only (filtered in-kernel).

pattern A function name, or a search pattern. Can include wildcards ("*"). If the -r option is used, can include regular expressions.

Count kernel and user stack traces for submit_bio(): # stackcount submit_bio

Count stacks with a delimiter for submit_bio(): # stackcount -d submit_bio

Count kernel stack trace only for submit_bio(): # stackcount -K submit_bio

Count user stack trace only for submit_bio(): # stackcount -U submit_bio

Count stack traces for ip_output(): # stackcount ip_output

Show symbol offsets: # stackcount -s ip_output

Show offsets and raw addresses (verbose): # stackcount -sv ip_output

Count stacks for kernel functions matching tcp_send*: # stackcount 'tcp_send*'

Same as previous, but using regular expressions: # stackcount -r '^tcp_send.*'

Output every 5 seconds, with timestamps: # stackcount -Ti 5 ip_output

Only count stacks when PID 185 is on-CPU: # stackcount -p 185 ip_output

Only count stacks for CPU 1: # stackcount -c 1 put_prev_entity

Count user stacks for dynamic heap allocations with malloc in PID 185: # stackcount -p 185 c:malloc

Count user stacks for thread creation (USDT tracepoint) in PID 185: # stackcount -p 185 u:pthread:pthread_create

Count stacks for context switch events using a kernel tracepoint: # stackcount t:sched:sched_switch

This summarizes unique stack traces in-kernel for efficiency, allowing it to trace a higher rate of function calls than methods that post-process in user space. The stack trace data is only copied to user space when the output is printed, which usually only happens once. The stack walking also happens in an optimized code path in the kernel thanks to the new BPF_STACK_TRACE table APIs, which should be more efficient than the manual walker in the eBPF tracer which older versions of this script used. With this in mind, call rates of < 10,000/sec would incur negligible overhead. Test before production use. You can also use funccount to get a handle on function call rates first. This is from bcc. https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

Linux Unstable - in development. Brendan Gregg, Sasha Goldshtein stacksnoop(8), funccount(8)