1*8d67ca89SAndroid Build Coastguard Worker## fdtrack 2*8d67ca89SAndroid Build Coastguard Worker 3*8d67ca89SAndroid Build Coastguard Worker[TOC] 4*8d67ca89SAndroid Build Coastguard Worker 5*8d67ca89SAndroid Build Coastguard Workerfdtrack is a file descriptor leak checker added to Android in API level 30. 6*8d67ca89SAndroid Build Coastguard Worker 7*8d67ca89SAndroid Build Coastguard Workerfdtrack consists of several parts: a set of hooks in bionic to register a 8*8d67ca89SAndroid Build Coastguard Workercallback that's invoked on file descriptor operations, a library that implements 9*8d67ca89SAndroid Build Coastguard Workera hook to perform and store backtraces for file descriptor creation, and 10*8d67ca89SAndroid Build Coastguard Workercode in frameworks to automatically enable it (and deliberately crash a process 11*8d67ca89SAndroid Build Coastguard Workerthat's leaking). 12*8d67ca89SAndroid Build Coastguard Worker 13*8d67ca89SAndroid Build Coastguard Worker### bionic hooks 14*8d67ca89SAndroid Build Coastguard Workerbionic provides a header in the `bionic_libc_platform_headers` header_lib at <[bionic/fdtrack.h](https://android.googlesource.com/platform/bionic/+/refs/heads/main/libc/platform/bionic/fdtrack.h)>. 15*8d67ca89SAndroid Build Coastguard WorkerRegister a callback with `android_fdtrack_compare_exchange_hook` to receive 16*8d67ca89SAndroid Build Coastguard Workercallbacks upon file descriptor creation and destruction. This function can be 17*8d67ca89SAndroid Build Coastguard Workercalled at any point in order to start capturing events, but be sure to properly 18*8d67ca89SAndroid Build Coastguard Workerhandle unbalanced closes. This callback may be called from an async signal safe 19*8d67ca89SAndroid Build Coastguard Workercontext, but not vfork (bionic tracks whether a thread is vforked, and chooses 20*8d67ca89SAndroid Build Coastguard Workernot to call callbacks when this is the case). 21*8d67ca89SAndroid Build Coastguard Worker 22*8d67ca89SAndroid Build Coastguard Worker### libfdtrack 23*8d67ca89SAndroid Build Coastguard Worker[libfdtrack](https://android.googlesource.com/platform/bionic/+/refs/heads/main/libfdtrack) 24*8d67ca89SAndroid Build Coastguard Workerimplements a library that uses libunwindstack to unwind and store fd creation backtraces. 25*8d67ca89SAndroid Build Coastguard Worker 26*8d67ca89SAndroid Build Coastguard Worker### frameworks 27*8d67ca89SAndroid Build Coastguard WorkerAs the name implies, `spawnFdLeakCheckThread` in SystemServer spawns a thread 28*8d67ca89SAndroid Build Coastguard Workerto monitor the number of open file descriptors every so often. 29*8d67ca89SAndroid Build Coastguard WorkerIf that passes a certain threshold, fdtrack is enabled. 30*8d67ca89SAndroid Build Coastguard WorkerIf it passes another threshold, the process is aborted. 31*8d67ca89SAndroid Build Coastguard WorkerThese thresholds are configurable via system properties: 32*8d67ca89SAndroid Build Coastguard Worker``` 33*8d67ca89SAndroid Build Coastguard Worker // Number of open file descriptors before fdtrack starts; default 1600. 34*8d67ca89SAndroid Build Coastguard Worker private static final String SYSPROP_FDTRACK_ENABLE_THRESHOLD = 35*8d67ca89SAndroid Build Coastguard Worker "persist.sys.debug.fdtrack_enable_threshold"; 36*8d67ca89SAndroid Build Coastguard Worker 37*8d67ca89SAndroid Build Coastguard Worker // Number of open file descriptors before aborting; default 3000. 38*8d67ca89SAndroid Build Coastguard Worker private static final String SYSPROP_FDTRACK_ABORT_THRESHOLD = 39*8d67ca89SAndroid Build Coastguard Worker "persist.sys.debug.fdtrack_abort_threshold"; 40*8d67ca89SAndroid Build Coastguard Worker 41*8d67ca89SAndroid Build Coastguard Worker // Number of seconds between open fd count checks; default 120s. 42*8d67ca89SAndroid Build Coastguard Worker private static final String SYSPROP_FDTRACK_INTERVAL = 43*8d67ca89SAndroid Build Coastguard Worker "persist.sys.debug.fdtrack_interval"; 44*8d67ca89SAndroid Build Coastguard Worker``` 45*8d67ca89SAndroid Build Coastguard WorkerNote that it's also possible to monitor the number of open file descriptors for 46*8d67ca89SAndroid Build Coastguard Workera given process from the shell. `adb shell watch ls -l /proc/<pid>/fd` will show 47*8d67ca89SAndroid Build Coastguard Workerthem (and you can choose your own update rate as an argument to `watch`). 48*8d67ca89SAndroid Build Coastguard Worker 49*8d67ca89SAndroid Build Coastguard Worker#### Using libfdtrack 50*8d67ca89SAndroid Build Coastguard Workerlibfdtrack registers its hook upon being loaded, so to start capturing 51*8d67ca89SAndroid Build Coastguard Workerbacktraces, `dlopen("libfdtrack.so", RTLD_GLOBAL)` is all that's needed. To dump 52*8d67ca89SAndroid Build Coastguard Workerits output to logcat, either use `fdtrack_dump`, or send the signal 53*8d67ca89SAndroid Build Coastguard Worker`BIONIC_SIGNAL_FDTRACK` (available from `<bionic/reserved_signals.h>`) to the 54*8d67ca89SAndroid Build Coastguard Workerprocess. If you wish to iterate through the results programmatically, 55*8d67ca89SAndroid Build Coastguard Worker`fdtrack_iterate` can be used (warning: this interface is currently unstable, 56*8d67ca89SAndroid Build Coastguard Workerdon't use it in code that can be used on multiple platform versions.) 57*8d67ca89SAndroid Build Coastguard Worker 58*8d67ca89SAndroid Build Coastguard Workerlibfdtrack adds a significant amount of overhead, so for processes that are 59*8d67ca89SAndroid Build Coastguard Workerlatency-critical like system_server, it's not feasible to always capture 60*8d67ca89SAndroid Build Coastguard Workerbacktraces. Instead, if you can detect that an fd leak is ongoing, turning on 61*8d67ca89SAndroid Build Coastguard Workerbacktraces for a while and then triggering a dump can be sufficient. 62*8d67ca89SAndroid Build Coastguard Workersystem_server [implements this approach](https://android.googlesource.com/platform/frameworks/base/+/679f3e4242b8e018eb7df90ef433f81088a64fff%5E%21/), 63*8d67ca89SAndroid Build Coastguard Workerspawning a thread that regularly checks the count of fds in the process, turns 64*8d67ca89SAndroid Build Coastguard Workeron fdtrack when it hits a threshold, and then aborts after another threshold. 65*8d67ca89SAndroid Build Coastguard WorkerThis dumps the output to logcat, which will be available in both the tombstone 66*8d67ca89SAndroid Build Coastguard Workerand logcat from bugreports. 67*8d67ca89SAndroid Build Coastguard Worker 68*8d67ca89SAndroid Build Coastguard Worker#### Implementation details 69*8d67ca89SAndroid Build Coastguard WorkerThere are multiple methods to unwind in Android: 70*8d67ca89SAndroid Build Coastguard Worker 71*8d67ca89SAndroid Build Coastguard Worker * libunwindstack 72*8d67ca89SAndroid Build Coastguard Worker * Pros 73*8d67ca89SAndroid Build Coastguard Worker * Primary method on the platform 74*8d67ca89SAndroid Build Coastguard Worker * Able to unwind through ART 75*8d67ca89SAndroid Build Coastguard Worker * Cons 76*8d67ca89SAndroid Build Coastguard Worker * Uses malloc internally: unsafe unless a separate allocator is 77*8d67ca89SAndroid Build Coastguard Worker statically-linked and steps are taken to prevent the unwind from being 78*8d67ca89SAndroid Build Coastguard Worker interrupted by a signal 79*8d67ca89SAndroid Build Coastguard Worker * Slow - infeasible to be used always in latency-sensitive processes 80*8d67ca89SAndroid Build Coastguard Worker * `android_unsafe_frame_pointer_chase` 81*8d67ca89SAndroid Build Coastguard Worker * Pros 82*8d67ca89SAndroid Build Coastguard Worker * Definitely async signal safe 83*8d67ca89SAndroid Build Coastguard Worker * Fast 84*8d67ca89SAndroid Build Coastguard Worker * Cons 85*8d67ca89SAndroid Build Coastguard Worker * Unable to unwind through ART because it doesn't maintain the frame pointer 86*8d67ca89SAndroid Build Coastguard Worker * Requires -fno-omit-frame-pointer to be used on all code being unwound 87*8d67ca89SAndroid Build Coastguard Worker through, which currently isn't the case on Android 88*8d67ca89SAndroid Build Coastguard Worker * Frame layout is a mess on 32-bit ARM: the ARM standard, clang, and GCC 89*8d67ca89SAndroid Build Coastguard Worker [all disagree](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92172) 90*8d67ca89SAndroid Build Coastguard Worker * Chasing the frame pointer will often result in multiple frames inside the 91*8d67ca89SAndroid Build Coastguard Worker same function 92*8d67ca89SAndroid Build Coastguard Worker 93*8d67ca89SAndroid Build Coastguard Workerlibfdtrack chooses to use libunwindstack for now, since unwinding through ART 94*8d67ca89SAndroid Build Coastguard Workeris critical to being useful for the initial user, system_server. 95