1 2--- 3title: "Skia Automated Testing" 4linkTitle: "Skia Automated Testing" 5 6--- 7 8 9Overview 10-------- 11 12Skia uses [Swarming](https://github.com/luci/luci-py/blob/main/appengine/swarming/doc/Design.md) 13to do the heavy lifting for our automated testing. It farms out tasks, which may 14consist of compiling code, running tests, or any number of other things, to our 15bots, which are virtual or real machines living in our local lab, Chrome Infra's 16lab, or in GCE. 17 18The [Skia Task Scheduler](http://go/skia-task-scheduler) determines what tasks 19should run on what bots at what time. See the link for a detailed explanation of 20how relative task priorities are derived. A *task* corresponds to a single 21Swarming task. A *job* is composed of a directed acyclic graph of one or more 22*tasks*. The job is complete when all of its component tasks have succeeded 23or is considered a failure when any of its component tasks fails. The scheduler 24may automatically retry tasks within its set limits. Jobs are not retried. 25Multiple jobs may share the same task, for example, tests on two different 26Android devices which use the same compiled code. 27 28Each Skia repository has an `infra/bots/tasks.json` file which defines the jobs 29and tasks for the repo. Most jobs will run at every commit, but it is possible 30to specify nightly and weekly jobs as well. For convenience, most repos also 31have a `gen_tasks.go` which will generate `tasks.json`. You will need to 32[install Go](https://golang.org/doc/install). From the repository root: 33 34 $ go run infra/bots/gen_tasks.go 35 36It is necessary to run `gen_tasks.go` every time it is changed or every time an 37[asset](https://skia.googlesource.com/skia/+/main/infra/bots/assets/README.md) 38has changed. There is also a test mode which simply verifies that the `tasks.json` 39file is up to date: 40 41 $ go run infra/bots/gen_tasks.go --test 42 43 44 45Try Jobs 46-------- 47 48Skia's trybots allow testing and verification of changes before they land in the 49repo. You need to have permission to trigger try jobs; if you need permission, 50ask a committer. After uploading your CL to [Gerrit](https://skia-review.googlesource.com/), 51you may trigger a try job for any job listed in `tasks.json`, either via the 52Gerrit UI, using `git cl try`, eg. 53 54 git cl try -B skia.primary -b Some-Tryjob-Name 55 56or using `bin/try`, a small wrapper for `git cl try` which helps to choose try jobs. 57From a Skia checkout: 58 59 bin/try --list 60 61You can also search using regular expressions: 62 63 bin/try "Test.*GTX660.*Release" 64 65 66Status View 67------------ 68 69The status view shows a table with tasks, grouped by test type and platform, 70on the X-axis and commits on the Y-axis. The cells are colored according to 71the status of the task for each commit: 72 73* green: success 74* orange: failure 75* purple: mishap (infrastructure issue) 76* black border, no fill: task in progress 77* blank: no task has started yet for a given revision 78 79Commits are listed by author, and the branch on which the commit was made is 80shown on the very left. A purple result will override an orange result. 81 82For more detail, you can click on an individual cell to get a summary of the 83task. You can also click one of the white bars at the top of each column to see 84a summary of recent tasks with the same name. 85 86The status page has several filters which can be used to show only a subset of 87task specs: 88 89* Interesting: Task specs which have both successes and failures within the 90 visible commit window. 91* Failures: Task specs which have failures within the visible commit window. 92* Comments: Task specs which have comments. 93* Failing w/o comment: task specs which have failures within the visible commit 94 window but have no comments. 95* All: Display all tasks. 96* Search: Enter a search string. Substrings and regular expressions may be 97 used, per the Javascript String Match() rules: 98 http://www.w3schools.com/jsref/jsref_match.asp 99 100<a name="adding-new-jobs"></a> 101Adding new jobs 102--------------- 103 104If you would like to add jobs to build or test new configurations, please file a 105[New Bot Request][new bot request]. 106 107If you know that the new jobs will need new hardware or you aren't sure which 108existing bots should run the new jobs, assign to jcgregorio. Once the Infra team 109has allocated the hardware, we will assign back to you to complete the process. 110 111Generally it's possible to copy an existing job and make changes to accomplish 112what you want. You will need to add the new job to 113[infra/bots/jobs.json][jobs json]. In some cases, you will need to make changes 114to recipes: 115 116* If there are new GN flags or compiler options: 117 [infra/bots/recipe_modules/build][build recipe module], probably default.py. 118* If there are modifications to dm flags: [infra/bots/recipes/test.py][test py] 119* If there are modifications to nanobench flags: 120 [infra/bots/recipes/perf.py][perf py] 121 122After modifying any of the above files, run `make train` in the infra/bots 123directory to update generated files. Upload the CL, then run `git cl try -B 124skia.primary -b <job name>` to run the new job. (After commit, the new job will 125appear in the PolyGerrit UI after the next successful run of the 126Housekeeper-Nightly-UpdateMetaConfig task.) 127 128[new bot request]: 129 https://bugs.chromium.org/p/skia/issues/entry?template=New+Bot+Request 130[jobs json]: https://skia.googlesource.com/skia/+/main/infra/bots/jobs.json 131[build recipe module]: 132 https://skia.googlesource.com/skia/+/refs/heads/main/infra/bots/recipe_modules/build/ 133[test py]: 134 https://skia.googlesource.com/skia/+/main/infra/bots/recipes/test.py 135[perf py]: 136 https://skia.googlesource.com/skia/+/main/infra/bots/recipes/perf.py 137 138 139Detail on Skia Tasks 140-------------------- 141 142[infra/bots/gen_tasks.go][gen_tasks] reads config files: 143 144* [infra/bots/jobs.json][jobs json] 145* [infra/bots/cfg.json][cfg json] 146* [infra/bots/recipe_modules/builder_name_schema/builder_name_schema.json][builder_name_schema] 147 148Based on each job name in jobs.json, gen_tasks decides which tasks to generate (process 149function). Various helper functions return task name of the direct dependencies of the job. 150 151In gen_tasks, tasks are specified with a TaskSpec. A TaskSpec specifies how to generate and trigger 152a Swarming task. 153 154Most Skia tasks run a recipe with Kitchen. The arguments to the kitchenTask function specify the 155most common parameters for a TaskSpec that will run a recipe. More info on recipes at 156[infra/bots/recipes/README.md][recipes README] and 157[infra/bots/recipe_modules/README.md][recipe_modules README]. 158 159The Swarming task is generated based on several parameters of the TaskSpec: 160 161* Isolate: specifies the isolate file. The isolate file specifies the files from the repo to place 162 on the bot before running the task. (For non-Kitchen tasks, the isolate also specifies the command 163 to run.) [More info][isolate user guide]. 164* Command: the command to run, if not specified in the Isolate. (Generally this is a boilerplate 165 Kitchen command that runs a recipe; see below.) 166* CipdPackages: specifies the IDs of CIPD packages that will be placed on the bot before running the 167 task. See infra/bots/assets/README.md for more info. 168* Dependencies: specifies the names of other tasks that this task depends upon. The outputs of those 169 tasks will be placed on the bot before running this task. 170* Dimensions: specifies what kind of bot should run this task. Ask Infra team for how to set this. 171* ExecutionTimeout: total time the task is allowed to run before it is killed. 172* IoTimeout: amount of time the task can run without printing something to stdout/stderr before it 173 is killed. 174* Expiration: Mostly ignored. If the task happens to be scheduled when there are no bots that can 175 run it, it will remain pending for this long before being canceled. 176 177If you need to do something more complicated, or if you are not sure how to add 178and configure the new jobs, please ask for help from borenet@, rmistry@ or jcgregorio@. 179 180[gen_tasks]: 181 https://skia.googlesource.com/skia/+/main/infra/bots/gen_tasks.go 182[cfg json]: 183 https://skia.googlesource.com/skia/+/main/infra/bots/cfg.json 184[builder_name_schema]: 185 https://skia.googlesource.com/skia/+/main/infra/bots/recipe_modules/builder_name_schema/builder_name_schema.json 186[recipes README]: 187 https://skia.googlesource.com/skia/+/main/infra/bots/recipes/README.md 188[recipe_modules README]: 189 https://skia.googlesource.com/skia/+/main/infra/bots/recipe_modules/README.md 190[isolate user guide]: 191 https://chromium.googlesource.com/infra/luci/luci-py/+/main/appengine/isolate/doc/client/Isolate-User-Guide.md 192 193