1# Testing 2 3Crosvm runs on a variety of platforms with a significant amount of platform-specific code. Testing 4on all the supported platforms is crucial to keep crosvm healthy. 5 6## Types of tests 7 8### Unit Tests 9 10Unit tests are your standard rust tests embedded with the rest of the code in `src/` and wrapped in 11a `#[cfg(test)]` attribute. 12 13Unit tests **cannot make any guarantees on the runtime environment**. Avoid doing the following in 14unit tests: 15 16- Avoid kernel features such as io_uring or userfaultfd, which may not be available on all kernels. 17- Avoid functionality that requires privileges (e.g. CAP_NET_ADMIN) 18- Avoid spawning threads or processes 19- Avoid accessing kernel devices 20- Avoid global state in unit tests 21 22This allows us to execute unit tests for any platform using emulators such as qemu-user-static or 23wine64. 24 25### Documentation tests 26 27Rust's 28[documentation tests](https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html) 29can be used to provide examples as part of the documentation that is verified by CI. 30 31Documentation tests are slow and not run as part of the usual workflows, but can be run locally 32with: 33 34```sh 35./tools/presubmit doc_tests 36``` 37 38### Integration tests 39 40Cargo has native support for 41[integration testing](https://doc.rust-lang.org/rust-by-example/testing/integration_testing.html). 42Integration tests are written just like unit tests, but live in a separate directory at `tests/`. 43 44Integration tests **guarantee that the test has privileged access to the test environment**. They 45are only executed when a device-under-test (DUT) is specified when running tests: 46 47```sh 48./tools/run_tests --dut=vm|host 49``` 50 51### End To End (E2E) tests 52 53End to end tests live in the `e2e_tests` crate. The crate provides a framework to boot a guest with 54crosvm and execut commands in the guest to validate functionality at a high level. 55 56E2E tests are executed just like integration tests. By giving 57[nextest's filter expressions](https://nexte.st/book/filter-expressions), you can run a subset of 58the tests. 59 60```sh 61# Run all e2e tests 62./tools/run_tests --dut=vm --filter-expr 'package(e2e_tests)' 63# Run e2e tests whose name contains the string 'boot'. 64./tools/run_tests --dut=vm --filter-expr 'package(e2e_tests) and test(boot)' 65``` 66 67### Downstream Product tests 68 69Each downstream product that uses crosvm is performing their own testing, e.g. ChromeOS is running 70high level testing of its VM features on ChromeOS hardware, while AOSP is running testing of their 71VM features on AOSP hardware. 72 73Upstream crosvm is not involved in these tests and they are not executed in crosvm CI. 74 75## Parallel test execution 76 77Crosvm tests are executed in parallel, each test case in its own process via 78[cargo nextest](http://nexte.st). 79 80This requires tests to be cautious about global state, especially integration tests which interact 81with system devices. 82 83If you require exclusive access to a device or file, you have to use 84[file-based locking](https://docs.rs/named-lock/latest/named_lock) to prevent access by other test 85processes. 86 87## Platforms tested 88 89The platforms below can all be tested using `tools/run_tests -p $platform`. The table indicates how 90these tests are executed: 91 92| Platform | Build | Unit Tests | Integration Tests | E2E Tests | 93| :-------------------------- | :---: | :-----------------------: | :---------------: | :-------: | 94| x86_64 (linux) | ✅ | ✅ | ✅ | ✅ | 95| aarch64 (linux) | ✅ | ✅ (qemu-user[^qemu-user]) | ✅ (qemu[^qemu]) | ❌ | 96| armhf (linux) | ✅ | ✅ (qemu-user[^qemu-user]) | ❌ | ❌ | 97| mingw64[^windows] (linux) | �� | �� (wine64) | ❌ | ❌ | 98| mingw64[^windows] (windows) | �� | �� | �� | ❌ | 99 100Crosvm CI will use the same configuration as `tools/run_tests`. 101 102## Debugging Tips 103 104Here are some tips for developing or/and debugging crosvm tests. 105 106### Enter a test VM to see logs 107 108When you run a test on a VM with `./tools/run_tests --dut=vm`, if the test fails, you'll see 109extracted log messages. To see the full messages or monitor the test process during the runtime, you 110may want to enter the test VM. 111 112First, enter the VM's shell and start printing the syslog: 113 114```console 115$ ./tools/dev_container # Enter the dev_container 116$ ./tools/x86vm shell # Enter the test VM 117crosvm@testvm-x8664:~$ journalctl -f 118# syslog messages will be printed... 119``` 120 121Then, open another terminal and run a test: 122 123```console 124$ ./tools/run_tests --dut=vm --filter-expr 'package(e2e_tests) and test(boot)' 125``` 126 127So you'll see the crosvm log in the first terminal. 128 129[^qemu-user]: qemu-aarch64-static or qemu-arm-static translate instructions into x86 and executes them on the 130 host kernel. This works well for unit tests, but will fail when interacting with platform 131 specific kernel features. 132 133[^qemu]: run_tests will launch a VM for testing in the background. This VM is using full system 134 emulation, which causes tests to be slow. Also not all aarch64 features are properly emulated, 135 which prevents us from running e2e tests. 136 137[^windows]: Windows builds of crosvm are a work in progress. Some tests are executed via wine64 on linux 138