memleak - Print a summary of outstanding allocations and their call stacks to detect memory leaks. Uses Linux eBPF/bcc. memleak [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND] [--combined-only] [--wa-missing-free] [-s SAMPLE_RATE] [-T TOP] [-z MIN_SIZE] [-Z MAX_SIZE] [-O OBJ] [INTERVAL] [COUNT] memleak traces and matches memory allocation and deallocation requests, and collects call stacks for each allocation. memleak can then print a summary of which call stacks performed allocations that weren't subsequently freed. When tracing a specific process, memleak instruments a list of allocation functions from libc, specifically: malloc, calloc, realloc, posix_memalign, valloc, memalign, pvalloc, aligned_alloc, and free. When tracing all processes, memleak instruments kmalloc/kfree, kmem_cache_alloc/kmem_cache_free, and also page allocations made by get_free_pages/free_pages. memleak may introduce significant overhead when tracing processes that allocate and free many blocks very quickly. See the OVERHEAD section below. This tool only works on Linux 4.6+. Stack traces are obtained using the new BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old. Kernel memory allocations are intercepted through tracepoints, which are available on Linux 4.7+. CONFIG_BPF and bcc.

-h Print usage message.

-p PID Trace this process ID only (filtered in-kernel). This traces libc allocator.

-t Print a trace of all allocation and free requests and results.

-a Print a list of allocations that weren't freed (and their sizes) in addition to their call stacks.

-o OLDER Print only allocations older than OLDER milliseconds. Useful to remove false positives. The default value is 500 milliseconds.

-c COMMAND Run the specified command and trace its allocations only. This traces libc allocator.

--combined-only Use statistics precalculated in kernel space. Amount of data to be pulled from kernel significantly decreases, at the cost of losing capabilities of time-based false positives filtering (-o).

--wa-missing-free Make up the action of free to alleviate misjudgments when free is missing.

-s SAMPLE_RATE Record roughly every SAMPLE_RATE-th allocation to reduce overhead.

-t TOP Print only the top TOP stacks (sorted by size). The default value is 10.

-z MIN_SIZE Capture only allocations that are larger than or equal to MIN_SIZE bytes.

-Z MAX_SIZE Capture only allocations that are smaller than or equal to MAX_SIZE bytes.

-O OBJ Attach to allocation functions in specified object instead of resolving libc. Ignored when kernel allocations are profiled.

INTERVAL Print a summary of outstanding allocations and their call stacks every INTERVAL seconds. The default interval is 5 seconds.

COUNT Print the outstanding allocations summary COUNT times and then exit.

Print outstanding kernel allocation stacks every 3 seconds: # memleak 3

Print user outstanding allocation stacks and allocation details for the process 1005: # memleak -p 1005 -a

Sample roughly every 5th allocation (~20%) of the call stacks and print the top 5 stacks 10 times before quitting. # memleak -s 5 --top=5 10

Run ./allocs and print outstanding allocation stacks for that process: # memleak -c "./allocs"

Capture only allocations between 16 and 32 bytes in size: # memleak -z 16 -Z 32

memleak can have significant overhead if the target process or kernel performs allocations at a very high rate. Pathological cases may exhibit up to 100x degradation in running time. Most of the time, however, memleak shouldn't cause a significant slowdown. You can use the -s switch to reduce the overhead further by capturing only every N-th allocation. The -z and -Z switches can also reduce overhead by capturing only allocations of specific sizes. Additionally, option --combined-only saves processing time by reusing already calculated allocation statistics from kernel. It's faster, but lacks information about particular allocations. Also, option --wa-missing-free makes memleak more accuracy in the complicated environment. To determine the rate at which your application is calling malloc/free, or the rate at which your kernel is calling kmalloc/kfree, place a probe with perf and collect statistics. For example, to determine how many calls to __kmalloc are placed in a typical period of 10 seconds: # perf probe '__kmalloc' # perf stat -a -e 'probe:__kmalloc' -- sleep 10 This is from bcc. https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

Linux Unstable - in development. Sasha Goldshtein