1VTS Kselftest Workflow 2====================== 3 4A Guide to Developing Kselftest for VTS/Android 5 6What is Kselftest? 7------------------ 8 9Kselftest is a collection of tests included within the Linux kernel repository 10at tools/testing/selftests. The tests cover a broad range of areas but total 11coverage (as of 4.14 anyway) is very spotty. Many of the tests do not apply to 12Android platforms. 13 14As of Jan 2023 there are 31 tests in kselftest which run on ARM in VTS. 1522 of these tests are executed in both 32-bit and 64-bit mode. There are many 16tests which are not enabled because they have dependencies which are not met on 17Android platforms. 18 19How is Kselftest Run in VTS? 20---------------------------- 21 22The Kselftest source is located at external/linux-kselftest in the Android 23tree. It is a set of files (tools/testing/selftests/) copied into its own 24repository from the Linux kernel repository. This is not an exact mirror of 25upstream; there are outstanding changes to kselftest for it to work on Android 26which have not yet been pushed upstream (contained in android/patches). In 27addition to the kselftest source there is also the VTS wrapper around it. This 28is located at test/vts-testcase/kernel/linux_kselftest. Some noteworthy 29directories/files: 30 31* `external/linux-kselftest/Android.bp`: Lists build rules for the kselftest modules built in Soong. 32* `external/linux-kselftest/android/`: Contains Android-specific files, aside from Android.[bp, mk] at top level. 33* `external/linux-kselftest/android/README`: Lists details on why particular tests are not enabled in Android. 34* `external/linux-kselftest/tools/testing/selftests`: Kselftest sources. 35* `test/vts-testcase/kernel/linux_kselftest/testcases/`: tests to run in VTS kselftest suites are 36specified in each of the test XML file, e.g, vts_linux_kselftest_arm_32.xml contains tests for 32 bit ARM platform. 37 38Running Kselftest through atest 39------------------------------- 40You can run Kselftest tests with atest, which handles all the setup and build steps. 41 42To run all 32 bit Kselftest tests: 43* `atest -a vts_linux_kselftest_arm_32` 44 45To run all 64 bit Kselftest tests: 46* `atest -a vts_linux_kselftest_arm_64` 47 48To run a single test: 49* `atest -a vts_linux_kselftest_arm_64:timers_valid-adjtimex_arm_64` 50 51Running Kselftest through VTS 52----------------------------- 53To run VTS kselftest through VTS, it must first be built. VTS is not device specific, 54you need not compile it specifically for the device you wish to run it on, assuming 55it is the same architecture. 56* `. build/envsetup.sh` 57* `lunch` 58* `make -j vts` 59 60Then open vts-tradefed and run the VTS tests (64 bit ARM platform as an example): 61* `vts-tradefed` 62* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64` 63 64If you regularly work with multiple devices it may be useful to specify the 65specific device you wish to run VTS on via the serial number: 66* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64 -s 000123456789` 67 68Or a specific test: 69* `vts-tf > run vts-kernel -m vts_linux_kselftest_arm_64 -t net_socket` 70 71Running Kselftest Directly 72-------------------------- 73 74Running kselftest tests within VTS can be quite cumbersome, especially if you 75are iterating a lot trying to debug something. Build and run kselftest tests 76faster by doing 77 78* `external/linux-kselftest$ mma` 79* `external/linux-kselftest$ adb sync data` 80 81The test cases will be located at `/data/nativetest{64,}/vts_linux_kselftest/`. 82 83Build just the kselftest test you care about with `m -j kselftest_<testname>` (listed in `external/linux-kselftest/Android.bp`). 84 85What Outstanding Issues Exist? 86------------------------------ 87 88The hotlist for kselftest bugs is 89[kselftest-todo](https://buganizer.corp.google.com/hotlists/745928). 90 91When you begin working on a kselftest bug please assign the bug to yourself so 92that others know it is being worked on. 93 94Testing x86_64 95-------------- 96 97It is not advisable to run kselftest tests directly on your host unless you are fully 98aware of what the tests will do and are okay with it. These tests may 99destabilize your box or cause data loss. If you need to run tests on an x86 100platform and are unsure if they are safe you should run them in emulation, in a 101virtualized environment, or on a dedicated development x86 platform. 102 103To run Kselftest tests for x86 platform, you can do: 104* `atest -a vts_linux_kselftest_x86_32` 105* `atest -a vts_linux_kselftest_x86_64` 106 107Sending Fixes Upstream 108---------------------- 109 110Kselftest is part of the upstream Linux kernel and as such you can use the 111get_maintainers.pl script there to see who to send patches to. Here’s an 112example using a commit already upstream: 113 114``` 115smuckle@smuckle:~/repos/linux$ git show 352909b49ba | scripts/get_maintainer.pl 116Shuah Khan <[email protected]> (maintainer:KERNEL SELFTEST FRAMEWORK,commit_signer:2/6=33%,authored:2/6=33%,added_lines:2/6=33%,removed_lines:3/6=50%) 117Thomas Gleixner <[email protected]> (commit_signer:2/6=33%) 118Greg Kroah-Hartman <[email protected]> (commit_signer:1/6=17%,authored:1/6=17%,added_lines:1/6=17%) 119"Radim Krčmář" <[email protected]> (commit_signer:1/6=17%) 120Ingo Molnar <[email protected]> (commit_signer:1/6=17%,commit_signer:1/1=100%) 121Andy Lutomirski <[email protected]> (authored:2/6=33%,added_lines:2/6=33%,removed_lines:2/6=33%,authored:1/1=100%,added_lines:500/500=100%) 122"Kirill A. Shutemov" <[email protected]> (authored:1/6=17%,added_lines:1/6=17%,removed_lines:1/6=17%) 123[email protected] (open list) 124[email protected] (open list:KERNEL SELFTEST FRAMEWORK) 125``` 126 127In summary patches should be sent to [email protected] and 128[email protected], Shuah Khan (kselftest maintainer), and anyone 129else who has touched the code in question. Standard Linux kernel coding style 130and patch rules apply. They should be checkpatch (scripts/checkpatch.pl in the 131kernel repository) clean and sent in plain text in canonical patch format. One 132easy way to do this is by using git format-patch and git send-email. 133 134Merging Fixes 135------------- 136 137When possible please merge fixes upstream first. Then cherrypick the change 138onto aosp/master in external/linux-kselftest. 139 140If your change cannot go upstream first for whatever reason, please commit a 141patch for it into external/linux-kselftest/android/patches. This allows easier 142tracking of the delta with upstream and streamlines upgrades to new kselftest 143releases. 144 145Updating Kselftest 146------------------ 147 148To merge in a new upstream version of kselftest: 1491. Do a git merge of the upstream tag with the "ours" policy, dropping all upstream changes. Do not commit yet. 1502. Delete tools/testing/selftests and replace it with a copy from the upstream kernel at the same tag as used above. 1513. Apply the patches in android/patches/, resolving conflicts as required. 1524. Test on all supported kernel versions, ensuring that any tests currently enabled in VTS do not generate new failures. 1535. Commit the merge. 154 155If the runtime of kselftest changes significantly be sure to update the runtime-hint and test-timeout parameters to VTS in 156test config files under 157`test/vts-testcase/kernel/linux_kselftest/testcases/`. 158 159Notes on Individual Tests 160------------------------- 161 162### bpf/ 163 164The bpf tests depend on the bpf and elf libraries, and several kernel 165config options that are not required by Android. 166 167### cpufreq/ 168 169Cpufreq is not required by Android. 170 171### cpu-hotplug/ 172 173Cpu hotplug is not required by Android. 174 175### filesystems/dnotify_test.c 176 177This test has an endless loop in it. It is not currently run as part of the 178kselftest suite. 179 180### firmware/ 181 182These tests depend on a kernel module enabled by CONFIG_TEST_FIRMWARE, which is 183not required by Android. 184 185### ftrace/ 186 187Ftrace is a debug feature and not required by Android. 188 189### gpio/ 190 191The gpio test requires a test kernel module and gpiolib, neither of which are 192required by Android. 193 194### ipc/ 195 196The ipc test requires CONFIG_EXPERT and CONFIG_CHECKPOINT_RESTORE be enabled, 197neither of which are required by Android. 198 199### kvm/ 200 201KVM is not required by Android. 202 203### lib/ 204 205The lib/ tests rely on kernel modules enabled by CONFIG_TEST_PRINTF, 206CONFIG_TEST_BITMAP, and CONFIG_PRIME_NUMBERS. None of these are required by 207Android. 208 209### locking/ 210 211The wait-wound mutex test depends on a kernel module enabled by 212CONFIG_WW_MUTEX_SELFTEST which is not required by Android. 213 214### media_tests/ 215 216The media_tests depend on /dev/mediaX and /dev/videoX files which may not be 217present. They also require manual inspection of dmesg and are not part of 218the normal kselftest suite. 219 220### membarrier/ 221 222The membarrier test depends on CONFIG_EXPERT and CONFIG_MEMBARRIER being 223enabled, neither of which are required by Android. 224 225### memfd/ 226 227These tests depend on FUSE which is not present in Android. 228 229### memory-hotplug/ 230 231Memory hotplug is not required by Android. 232 233### mount/ 234 235The mount test depends on CONFIG_USER_NS which is not required by Android. 236 237### mqueue/ 238 239These tests depend on -lrt and -lpopt which are not present in Android. 240 241### net/ 242 243The test_bpf.sh test requires a kernel module enabled by CONFIG_TEST_BPF. 244The netdevice.sh test operates on an ethernet interface. 245The psock_fanout test depends on CONFIG_BPF_SYSCALL which is not required on 246Android. 247 248### netfilter/ 249 250These tests depend on CONFIG_NET_NS and CONFIG_NF_TABLES_INET, neither of which 251are required by Android. 252 253### nsfs/ 254 255These tests depend on CONFIG_USER_NS and CONFIG_PID_NS, neither of which are 256required by Android. 257 258### ntb/ 259 260NTB (non-transparent bridge) is not required by Android. 261 262### pstore/ 263 264Pstore is recommended, not required, for Android. 265 266### ptp/ 267 268PTP support is not required by Android. 269 270### rseq/ 271 272The rseq system call, enabled by CONFIG_RSEQ, is not required by Android. 273 274### sigaltstack/ 275 276This is a test for functionality that is not available in bionic (swapcontext()). 277 278### static_keys/ 279 280This test depends on a kernel module enabled by CONFIG_TEST_STATIC_KEYS. This 281kernel module is not required by Android. 282 283### sync/ 284 285These tests verify the legacy sync API which is deprecated on Android. 286 287### sysctl 288 289This test depends on a kernel module enabled by CONFIG_TEST_SYSCTL. This kernel 290module is not required by Android. 291 292### timers/ 293 294The adjtick test adjusts the system time and takes 2 minutes. 295 296The change_skew test takes too long to run as part of VTS. 297 298The clocksource_switch test takes too long to run as part of VTS and also tries 299all clocksources. 300 301The freq_step test relies on clocks having a greater precision than what may be 302available. 303 304The leap-a-day test repeatedly modifies the system date in order to test leap 305second behavior. This may disrupt the target environment. 306 307The leapcrash test adjusts the system date and takes several minutes to run. 308 309The mqueue_lat test relies on mq_* calls which are not present on Android. 310 311The rtctest_setdate test modifies the RTC's time and date which may disrupt the 312target environment. 313 314The set-2038 test repeatedly sets the time in order to test edge cases. This 315may disrupt the target enviornment. The test also takes a long time. 316 317The set-tz test sets the timezone which may disrupt the target environment. 318 319The skew_consistency test takes too long to run as part of VTS. 320 321### uevent/ 322 323This test depends on CONFIG_USER_NS which is not required by Android. 324 325### user/ 326 327This test depends on a kernel module enabled by CONFIG_TEST_USER_COPY. This 328kernel module is not required by Android. 329 330### vm/ 331 332The hugepage-mmap, hugepage-shm, compaction, and map_hugetlb tests rely on 333hugetlbfs/hugetlb page support which is not required by Android. 334 335### watchdog/ 336 337The watchdog test depends on CONFIG_WATCHDOG which is not required by Android. 338 339### zram/ 340 341The zram test depends on CONFIG_ZSMALLOC and CONFIG_ZRAM which are not required 342by Android.