1# Running tests 2 3The testing strategy for Perfetto is rather complex due to the wide variety 4of build configurations and embedding targets. 5 6Common test targets (all platforms / checkouts): 7 8`perfetto_unittests`: 9Platform-agnostic unit-tests. 10 11`perfetto_integrationtests`: 12End-to-end tests, involving the protobuf-based IPC transport and ftrace 13integration (Linux/Android only). 14 15`perfetto_benchmarks`: 16Benchmarks tracking the performance of: (i) trace writing, (ii) trace readback 17and (iii) ftrace raw pipe -> protobuf translation. 18 19Running tests on Linux / MacOS 20------------------------------ 21 22```bash 23tools/ninja -C out/default perfetto_{unittests,integrationtests,benchmarks} 24out/default/perfetto_unittests --gtest_help 25``` 26 27`perfetto_integrationtests` requires that the ftrace debugfs directory is 28is readable/writable by the current user on Linux: 29```bash 30sudo chown -R $USER /sys/kernel/debug/tracing 31``` 32 33Running tests on Android 34------------------------ 351A) Connect a device through `adb` 361B) Start the build-in emulator (supported on Linux and MacOS): 37 38```bash 39tools/install-build-deps --android 40tools/run_android_emulator & 41``` 42 432) Run the tests (either on the emulator or physical device): 44 45```bash 46tools/run_android_test out/default perfetto_unittests 47``` 48 49Continuous testing 50------------------ 51Perfetto is tested in a variety of locations: 52 53**Perfetto CI**: https://ci.perfetto.dev/ 54Builds and runs perfetto_{unittests,integrationtests,benchmarks} from the 55standalone checkout. Benchmarks are ran in a reduced form for smoke testing. 56See [this doc](/docs/design-docs/continuous-integration.md) for more details. 57 58**Android CI** (see go/apct and go/apct-guide): 59runs only `perfetto_integrationtests` 60 61**Android presubmits (TreeHugger)**: 62Runs before submission of every AOSP CL of `external/perfetto`. 63 64**Android CTS** (Android test suite used run to ensure API compatibility): 65Rolling runs internally. 66 67Note that Perfetto CI uses the standalone build system and the others build as 68part of the Android tree. 69 70Unit tests 71---------- 72Unit tests exist for most of the code in Perfetto on the class level. They 73ensure that each class broadly works as expected. 74 75Unit tests are currently ran on ci.perfetto.dev and build.chromium.org. 76Running unit tests on APCT and Treehugger is WIP. 77 78Integration tests 79----------------- 80Integration tests ensure that subsystems (importantly ftrace and the IPC layer) 81and Perfetto as a whole is working correctly end-to-end. 82 83There are two configurations in which integration tests can be run: 84 85**1. Production mode** (Android-only) 86This mode assumes that both the tracing service (`traced`) and the OS probes 87service (`traced_probes`) are already running. In this mode the test enables 88only the consumer endpoint and tests the interaction with the production 89services. This is the way our Android CTS and APCT tests work. 90 91**2. Standalone mode**: 92Starting up the daemons in the test itself and then testing against them. 93This is how standalone builds are tested. This is the only supported way to 94run integration tests on Linux and MacOS. 95 96Trace Processor diff tests 97----------------- 98Trace processor is mainly tested using so called "diff tests". 99 100For these tests, trace processor parses a known trace and executes a query 101string or file. The output of these queries is then compared (i.e. "diff"ed) against 102an expected output file and discrepencies are highlighted. 103 104Similar diff tests are also available when writing metrics - instead of a 105query, the metric name is used and the expected output string contains 106the expected result of computing the metric. 107 108These tests (for both queries and metrics) can be run as follows: 109```bash 110tools/ninja -C <out directory> 111tools/diff_test_trace_processor.py <out directory>/trace_processor_shell 112``` 113 114TIP: Query diff tests are expected to only have a single query which produces 115output in the whole file (usually at the end). Calling 116`SELECT RUN_METRIC('metric file')` can trip up this check as this query 117generates some hidden output. To address this issue, if a query only has 118column is named `suppress_query_output`, even if it has output, this will 119be ignored (for example, 120`SELECT RUN_METRIC('metric file') as suppress_query_output`) 121 122UI pixel diff tests 123----------------- 124The pixel tests are used to ensure core user journeys work by verifying they 125are the same pixel to pixel against a golden screenshot. They use a headless 126chrome to load the webpage and take a screenshot and compare pixel by pixel a 127golden screenshot. You can run these tests by using `ui/run-integrationtests`. 128 129 130These test fail when a certain number of pixels are different. If these tests 131fail, you'll need to investigate the diff and determine if its intentional. If 132its a desired change you will need to update the screenshots on a linux machine 133to get the CI to pass. You can update them by generating and uploading a new 134baseline (this requires access to a google bucket through gcloud which only 135googlers have access to, googlers can install gcloud 136[here](https://g3doc.corp.google.com/cloud/sdk/g3doc/index.md#installing-and-using-the-cloud-sdk)). 137 138``` 139ui/run-integrationtests --rebaseline 140tools/test_data upload 141``` 142 143Once finished you can commit and upload as part of your CL to cause the CI to 144use your new screenshots. 145 146NOTE: If you see a failing diff test you can see the pixel differences on the CI 147by using a link ending with `ui-test-artifacts/index.html`. Report located on 148that page contains changed screenshots as well as a command to accept the 149changes if these are desirable. 150 151Android CTS tests 152----------------- 153CTS tests ensure that any vendors who modify Android remain compliant with the 154platform API. 155 156These tests include a subset of the integration tests above as well as adding 157more complex tests which ensure interaction between platform (e.g. Android apps 158etc.) and Perfetto is not broken. 159 160The relevant targets are `CtsPerfettoProducerApp` and `CtsPerfettoTestCases`. Once these are built, the following commands should be run: 161 162```bash 163adb push $ANDROID_HOST_OUT/cts/android-cts/testcases/CtsPerfettoTestCases64 /data/local/tmp/ 164adb install -r $ANDROID_HOST_OUT/cts/android-cts/testcases/CtsPerfettoProducerApp.apk 165``` 166 167Next, the app named `android.perfetto.producer` should be run on the device. 168 169Finally, the following command should be run: 170 171```bash 172adb shell /data/local/tmp/CtsPerfettoTestCases64 173``` 174 175{#chromium} Chromium waterfall 176------------------ 177Perfetto is constantly rolled into chromium's //third_party/perfetto via 178[this autoroller](https://autoroll.skia.org/r/perfetto-chromium-autoroll). 179 180The [Chromium CI](https://build.chromium.org) runs the `perfetto_unittests` 181target, as defined in the [buildbot config][chromium_buildbot]. 182 183You can also test a pending Perfetto CL against Chromium's CI / TryBots 184before submitting it. This can be useful when making trickier API changes or to 185test on platforms that the Perfetto CI doesn't cover (e.g. Windows, MacOS), 186allowing you to verify the patch before you submit it (and it then eventually 187auto-rolls into Chromium). 188 189To do this, first make sure you have uploaded your Perfetto patch to the 190Android Gerrit. Next, create a new Chromium CL that modifies Chromium's 191`//src/DEPS` file. 192 193If you recently uploaded your change, it may be enough to modify the git commit 194hash in the `DEPS` entry for `src/third_party/perfetto`: 195 196``` 197 'src/third_party/perfetto': 198 Var('android_git') + '/platform/external/perfetto.git' + '@' + '8fe19f55468ee227e99c1a682bd8c0e8f7e5bcdb', 199``` 200 201Replace the git hash with the commit hash of your most recent patch set, which 202you can find in gerrit next to the active patch set number. 203 204Alternatively, you can add `hooks` to patch in the pending CL on top of 205Chromium's current third_party/perfetto revision. For this, add the following 206entries to the `hooks` array in Chromium's `//src/DEPS` file, modifying the 207`refs/changes/XX/YYYYYYY/ZZ` to the appropriate values for your gerrit change. 208You can see these values when pressing the "Download" button in gerrit. You can 209also use this method to patch in multiple Perfetto changes at once by 210adding additional `hooks` entries. [Here][chromium_cl]'s an example CL. 211 212``` 213 { 214 'name': 'fetch_custom_patch', 215 'pattern': '.', 216 'action': [ 'git', '-C', 'src/third_party/perfetto/', 217 'fetch', 'https://android.googlesource.com/platform/external/perfetto', 218 'refs/changes/XX/YYYYYYY/ZZ', 219 ], 220 }, 221 { 222 'name': 'apply_custom_patch', 223 'pattern': '.', 224 'action': ['git', '-C', 'src/third_party/perfetto/', 225 '-c', 'user.name=Custom Patch', '-c', 'user.email[email protected]', 226 'cherry-pick', 'FETCH_HEAD', 227 ], 228 }, 229``` 230 231If you'd like to test your change against the SDK build of Chrome, you 232can add `Cq-Include-Trybots:` lines for perfetto SDK trybots to the change 233description in gerrit (this won't be needed once Chrome's migration to the 234SDK is complete, see [tracking bug][sdk_migration_bug]): 235 236``` 237Cq-Include-Trybots: luci.chromium.try:linux-perfetto-rel 238Cq-Include-Trybots: luci.chromium.try:android-perfetto-rel 239Cq-Include-Trybots: luci.chromium.try:mac-perfetto-rel 240Cq-Include-Trybots: luci.chromium.try:win-perfetto-rel 241``` 242 243[chromium_buildbot]: https://cs.chromium.org/search/?q=perfetto_.*tests+f:%5Esrc/testing.*json$&sq=package:chromium&type=cs 244[chromium_cl]: https://chromium-review.googlesource.com/c/chromium/src/+/2030528 245[sdk_migration_bug]: https://crbug.com/1006541 246