1# Testing on bots 2 3Sometimes a failure happens consistently (or flakily) on bots but is difficult to reproduce locally 4due to a different platform, driver version, etc. The same build can be triggered on a matching bot 5with additional arguments using the following steps. Triggering swarming tasks from a local build 6can also sometimes be useful (see [`scripts/trigger.py`](../scripts/trigger.py)). 7 8## Navigate to the shard 9 10 11 12## Note the task dimensions as well as the "CAS inputs" identifier 13 14 15 16* Task dimensions is a filter that limits which bots in the swarming pool will pick up the task. For 17 example, here we can limit to the same OS and GPU with `-d os=Windows-10` and 18 `-d gpu=8086:9bc5-31.0.101.2127` (note: this numeric GPU id encodes both the vendor and the 19 specific driver version) 20 21## Find additional args required to repro 22 23The failure may or may not repro in isolation. Usually the test log will contain `--gtest_filter=` 24with the batch that was used when the failure occurred: 25 26 27 28* If not reproducible in isolation, it's usually easiest to start with the same batch to confirm the 29 failure is reproduced and then trim the list down. 30 31* Sometimes additional args are required (can be found in logs of the original task). 32 33## Triggering a swarming task 34 35You can trigger swarming tasks directly using `tools/luci-go/swarming trigger` from an ANGLE 36checkout. 37 38ACLs: ANGLE realm (e.g. `angle:try`) is guarded by 39https://chrome-infra-auth.appspot.com/auth/groups/project-angle-owners. If that shows 40`PermissionDenied`, you could also try `chromium:try`. 41 42For example, trigger that reproduced the failure in the example above - filter had to include the 43failing test and the test that ran right before it: 44 45``` 46% tools/luci-go/swarming trigger \ 47 -digest=e11fb5a14596dce84e86a4776d65c5da26acda8e5b04257988cf2fa8ac4c5630/399 \ 48 -realm angle:try \ 49 -priority=20 \ 50 -server=https://chromium-swarm.appspot.com \ 51 -d os=Windows-10 \ 52 -d pool=chromium.tests.gpu \ 53 -d cpu=x86-64 \ 54 -d gpu=8086:9bc5-31.0.101.2127 \ 55 -service-account=chromium-tester@chops-service-accounts.iam.gserviceaccount.com \ 56 -env=ISOLATED_OUTDIR=\${ISOLATED_OUTDIR} \ 57 -relative-cwd=out/Release_x64 \ 58 -- vpython3 ../../testing/test_env.py \ 59 ./angle_end2end_tests.exe \ 60 --isolated-script-test-output=\${ISOLATED_OUTDIR}/output.json \ 61 --gtest_filter=EGLDisplayTest.InitializeMultipleTimesInDifferentThreads/ES2_D3D11_NoFixture:EGLPresentPathD3D11.ClientBufferPresentPathFast/ES2_D3D11_NoFixture 62``` 63 64Additional notes: 65 66* It occasionally matters that bots run with `--test-launcher-bot-mode` - this sets `mBotMode=true` 67 in ANGLE harness and enables running multiple windows in parallel (however, on some bots 68 multi-processing is deactivated due to flakes, in which case you can find `--max-processes` arg 69 in the logs). Naturally, multiple windows are not supported on Android bots. If the failure 70 you're investigating is on a platform which runs with multi-processing, you might want to try 71 experimenting with these flags. 72 73* See [`scripts/trigger.py`](../scripts/trigger.py) for triggering tasks from local builds - it 74 first produces a CAS digest and then triggers a task using swarming trigger similar to the 75 command above. 76 77* CAS digests from bot builds can also be useful, e.g. by uploading a change and triggering a 78 builder to get a build on a platform you may not have access to, or by taking a digest of a 79 previous CI failure. 80 81* `-relative-cwd` and binary can be figured out by clicking on "CAS inputs" and inspecting `out`; 82 this can also be found in task logs. 83
