1# Handling Test Failures 2 3ANGLE is tested by a number of test suites, including by the ANGLE CI and Try testers, and by the 4Chromium FYI testers. See pointers and explanations in the 5[ANGLE Wrangling](https://chromium.googlesource.com/angle/angle/+/refs/heads/main/infra/ANGLEWrangling.md) documentation. 6 7We run a large number of tests for each ANGLE CL, both in ANGLE standalone and Chromium 8configurations, both pre- and post- commit. Some tests will fail, crash, or timeout. If these 9cannot be addressed in a timely manner, file a bug and update test expectations files. Timeliness 10depends on the context. For example, a Wrangler trying to unblock an AutoRoller will typically 11suppress failures immediately; where a developer will typically delay landing their CL in favor of 12fixing test failures. 13 14## Handling a Vulkan Validation Layer error 15 16Many tests are run with an option that enables the Vulkan Validation Layers (sometimes referred to 17as VVL). Validation errors will cause an otherwise-passing test to fail. 18 19The [vulkan-deps into ANGLE AutoRoller](https://autoroll.skia.org/r/vulkan-deps-angle-autoroll) 20updates ANGLE to the top-of-tree (ToT) upstream Vulkan tools and SDK. Sometimes validation errors 21are the result of bugs in the Vulkan Validation Layers, sometimes because of bugs in ANGLE. 22Therefore, investigate the cause of the error and determine if it's an ANGLE bug or a Vulkan 23Validation Layer bug. For Vulkan Validation Layer bugs, file an 24[upstream bug](https://github.com/KhronosGroup/Vulkan-ValidationLayers/issues/new), and 25suppress the error. The ANGLE Wrangler will also suppress a validation error when the 26`vulkan-deps` AutoRoller introduces a new validation error. The ANGLE Wrangler isn't expected to 27resolve the error or diagnose an upstream bug (but it is welcome as extra credit). 28 29Handle a validation error by doing the following: 30 311. [File an ANGLE bug](http://anglebug.com/new). If this is an active Wrangler issue, set the Label 32 `Hotlist-Wrangler` on the bug. 332. Add the VVL error tag to the 34 [kSkippedMessages](https://chromium.googlesource.com/angle/angle.git/+/8f8ca06dfb903fcc8517c69142c46c05e618f40d/src/libANGLE/renderer/vulkan/RendererVk.cpp#129) 35 array in `vk_renderer.cpp` file. Follow the pattern for adding a comment with the associated bug 36 in the line above the VVL tag. 37 38 39## dEQP test expectations 40 41There are a set of [dEQP](dEQP.md) expectations files in the 42[src/tests/deqp_support](../src/tests/deqp_support) directory. Notice the format of a line and 43your choices for OS, driver, etc. This is described in the directory's 44[README.md](../src/tests/deqp_support/README.md) file. This includes: 45- `FLAKY` - For rare flakes 46- `FAIL` - For a test that flakes often or persistently fails 47- `SKIP` - For a test that crashes 48- `TIMEOUT` - For a test that is very slow and may timeout 49 50 51## angle_end2end_tests expectations 52 53These expectations all live in the 54[angle_end2end_tests_expectations.txt](../src/tests/angle_end2end_tests_expectations.txt) file. The file format 55is the same as for the dEQP expectations. However, `FAIL` and `FLAKY` are not valid, and so the choices are: 56- `SKIP` - For a test that fails or crashes 57- `TIMEOUT` - For a test that is very slow and may timeout 58 59 60## Rubber stamping expectations CLs 61 62CLs that only change dEQP or angle_end2end_tests expectations don't require a human reviewer and can be 63[Rubber Stamped](https://chromium.googlesource.com/infra/infra/+/refs/heads/main/go/src/infra/appengine/rubber-stamper/README.md). 64The procedure is: 651. Add [email protected] as the reviewer on the expectations CL. Please also CC a human reviewer for TBR. 662. After 1 minute, Rubber Stamper should set "Bot-Commit: +1" on the CL. 673. Now you should be able to "Commit-Queue: +2" your CL. 68 69 70## WebGL conformance test expectations 71 72The expectations files are hosted in the `chromium/src` repository under 73[content/test/gpu/gpu_tests/test_expectations](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/content/test/gpu/gpu_tests/test_expectations/). 74Note that this is not included in local ANGLE-only source tree. 75 76The format of the file, including the different tags, is documented at the top of the file. This 77includes the following results: 78 79- `RetryOnFailure` - For a test that rarely flakes 80- `Failure` - For a test that fails consistently or flakes often 81- `Skip` - For a test that causes catastrophic failures (e.g. ends an entire test run, causes a bot 82 to BSoD); `Skip` should be used very sparingly 83 84You will need to contact an OWNER of the file to +1 your CL. 85 86You have two options for creating a CL to the expectations files: 87 881. For trivial edits, you can edit the expectations files via 89 [Chromium Code Search](https://source.chromium.org/chromium/chromium/src/+/main:content/test/gpu/gpu_tests/test_expectations/): 90 - In the browser, press the `Edit code` button. This will bring up a new browser window/tab, 91 in an editor mode. 92 - Edit the expecations and then press the `Create change` (or `Update change` button if you 93 need to change your CL later), which will create a CL. 942. Otherwise please [check out the code](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/get_the_code.md) 95 and [upload a CL](https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/contributing.md#Creating-a-change) 96 97 98## "internal swarming failure" on one of the test shards 99 100This might be caused by a bad bot, clicking on link "Bot assigned to task" will show recent tasks which might look bad 101(for example, lots of BOT_DIED in [crbug.com/1350556](https://crbug.com/1350556)). The best course of action in this case is to 102stop the bot by clicking the "stop the bot gracefully" button which will result in a task that looks like 103"Terminate build117-a9", then open a bug in the Infra>Labs component. 104Googlers, see also: [go/gpu-pixel-wrangler-how-to#manually-taking-a-machine-out-of-the-swarming-pool](http://go/gpu-pixel-wrangler-how-to#manually-taking-a-machine-out-of-the-swarming-pool) 105 106 107# Requesting access to the Commit Queue and issue tracker 108 109If you need to request access for a user (or yourself) please ask a project owner to help. Googlers, email 110[email protected]. External users, find a couple owners in the [OWNERS](../OWNERS) file and send a 111request via email. 112 113If you are an ANGLE owner and you want to grant a user access: 114 115 * For the Monorail issue tracker, add users via the ["People" administration console][MonorailPeople]. 116 * For the ANGLE Commit Queue (CQ), grant try-job access via the [Chrome Infra Auth administration console][ChromeAdmin]. 117 * The admin console also has a separate group for [CQ access][CQAccessAdmin]. 118 119[MonorailPeople]: https://bugs.chromium.org/p/angleproject/people/list 120[ChromeAdmin]: https://chrome-infra-auth.appspot.com/auth/groups/project-angle-tryjob-access 121[CQAccessAdmin]: https://chrome-infra-auth.appspot.com/auth/groups/project-angle-committers 122