1# SafetyNet - Performance regression detection for PDFium 2 3[TOC] 4 5This document explains how to use SafetyNet to detect performance regressions 6in PDFium. 7 8## Comparing performance of two versions of PDFium 9 10safetynet_compare.py is a script that compares the performance between two 11versions of pdfium. This can be used to verify if a given change has caused 12or will cause any positive or negative changes in performance for a set of test 13cases. 14 15The supported profilers are exclusive to Linux, so for now this can only be run 16on Linux. 17 18An illustrative example is below, comparing the local code version to an older 19version. Positive % changes mean an increase in time/instructions to run the 20test - a regression, while negative % changes mean a decrease in 21time/instructions, therefore an improvement. 22 23``` 24$ testing/tools/safetynet_compare.py ~/test_pdfs --branch-before beef5e4 25================================================================================ 26 % Change Time after Test case 27-------------------------------------------------------------------------------- 28 -0.1980% 45,703,820,326 ~/test_pdfs/PDF Reference 1-7.pdf 29 -0.5678% 42,038,814 ~/test_pdfs/Page 24 - PDF Reference 1-7.pdf 30 +0.2666% 10,983,158,809 ~/test_pdfs/Rival.pdf 31 +0.0447% 10,413,890,748 ~/test_pdfs/dynamic.pdf 32 -7.7228% 26,161,171 ~/test_pdfs/encrypted1234.pdf 33 -0.2763% 102,084,398 ~/test_pdfs/ghost.pdf 34 -3.7005% 10,800,642,262 ~/test_pdfs/musician.pdf 35 -0.2266% 45,691,618,789 ~/test_pdfs/no_metadata.pdf 36 +1.4440% 38,442,606,162 ~/test_pdfs/test7.pdf 37 +0.0335% 9,286,083 ~/test_pdfs/testbulletpoint.pdf 38================================================================================ 39Test cases run: 10 40Failed to measure: 0 41Regressions: 0 42Improvements: 2 43``` 44 45### Usage 46 47Run the safetynet_compare.py script in testing/tools to perform a comparison. 48Pass one or more paths with test cases - each path can be either a .pdf file or 49a directory containing .pdf files. Other files in those directories are 50ignored. 51 52The following comparison modes are supported: 53 541. Compare uncommitted changes against clean branch: 55```shell 56$ testing/tools/safetynet_compare.py path/to/pdfs 57``` 58 592. Compare current branch with another branch or commit: 60```shell 61$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before another_branch 62$ testing/tools/safetynet_compare.py path/to/pdfs --branch-before 1a3c5e7 63``` 64 653. Compare two other branches or commits: 66```shell 67$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before yet_another_branch 68$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after 1a3c5e7 --branch-before 0b2d4f6 69$ testing/tools/safetynet_compare.py path/to/pdfs --branch-after another_branch --branch-before 0b2d4f6 70``` 71 724. Compare two build flag configurations: 73```shell 74$ gn args out/BuildConfig1 75$ gn args out/BuildConfig2 76$ testing/tools/safetynet_compare.py path/to/pdfs --build-dir out/BuildConfig2 --build-dir-before out/BuildConfig1 77``` 78 79safetynet_compare.py takes care of checking out the appropriate branch, building 80it, running the test cases and comparing results. 81 82### Profilers 83 84safetynet_compare.py uses callgrind as a profiler by default. Use --profiler 85to specify another one. The supported ones are: 86 87#### perfstat 88 89Only works on Linux. 90Make sure you have perf by typing in the terminal: 91```shell 92$ perf 93``` 94 95This is a fast profiler, but uses sampling so it's slightly inaccurate. 96Expect variations of up to 1%, which is below the cutoff to consider a 97change significant. 98 99Use this when running over large test sets to get good enough results. 100 101#### callgrind 102 103Only works on Linux. 104Make sure valgrind is installed: 105```shell 106$ valgrind 107``` 108 109This is a slow and accurate profiler. Expect variations of around 100 110instructions. However, this takes about 50 times longer to run than perf stat. 111 112Use this when looking for small variations (< 1%). 113 114One advantage is that callgrind can generate `callgrind.out` files (by passing 115--output-dir to safetynet_compare.py), which contain profiling information that 116can be analyzed to find the cause of a regression. KCachegrind is a good 117visualizer for these files. 118 119#### none 120 121Run without any profiler, giving a performance score of 1 always. useful for 122running image comparisons or debugging the script. 123 124### Common Options 125 126Arguments commonly passed to safetynet_compare.py. 127 128* --profiler: described above. 129* --build-dir: this specified the build config with a relative path from the 130pdfium src directory to the build directory. Defaults to out/Release. 131* --output-dir: where to place the profiling output files. These are 132callgrind.out.[test_case] files for callgrind, perfstat does not produce them. 133By default they are not written. 134* --case-order: sort test case results according to this metric. Can be "after", 135"before", "ratio" and "rating". If not specified, sort by path. 136* --this-repo: use the repository where the script is instead of checking out a 137temporary one. This is faster and does not require downloads. Although it 138restores the state of the local repo, if the script is killed or crashes the 139uncommitted changes can remain stashed and you may be on another branch. 140 141### Other Options 142 143Most of the time these don't need to be used. 144 145* --build-dir-before: if comparing different build dirs (say, to test what a 146flag flip does), specify the build dir for the “before” branch here and the 147build dir for the “after” branch with --build-dir. 148* --interesting-section: only the interesting section should be measured instead 149of all the execution of the test harness. This only works in debug, since in 150release the delimiters are stripped out. This does not work to compare branches 151that don’t have the callgrind delimiters, as it would otherwise be unfair to 152compare a whole run vs the interesting section of another run. 153* --machine-readable: output a json with the results that is easier to read by 154code. 155* --num-workers: how many workers to use to parallelize test case runs. Defaults 156to # of CPUs in the machine. 157* --threshold-significant: highlight differences that exceed this value. 158Defaults to 0.02. 159* --tmp-dir: directory in which temporary repos will be cloned and downloads 160will be cached, if --this-repo is not enabled. Defaults to /tmp. 161 162## Setup a nightly job 163 164Create a separate checkout of pdfium in a new directory, for example `~/job`. 165The safetynet_job.py script will run from this directory. This checkout needs to 166be `git pull`'ed when there are changes to the SafetyNet scripts, but otherwise 167it can be left alone. 168 169Create a directory to contain the job results, for example `~/job_results`. In 170each run, a `.log` file with the results will be written to this directory and a 171subdirectory will be created with the other artifacts. 172 173Setup a cron job to run safetynet_job.py nightly. The example below runs it at 1741:42 AM, over the corpus in two directories: `~/pdf_samples/thousand_pdfs` and 175`~/pdf_samples/i18n` 176 177```shell 178@ crontab -e 17942 1 * * * bash -lc '~/job/pdfium/testing/tools/safetynet_job.py ~/job_results ~/pdf_samples/thousand_pdfs ~/pdf_samples/i18n --output-to-log >> ~/job_results/cron_nightly.log 2>&1' 180``` 181 182The first time the job runs, it will just create a checkpoint as 183`~/job_results/last_revision_covered`. From then on, since a checkpoint is 184available, each run will compare performance with the last checkpoint and update 185the checkpoint. 186 187## Run image comparison 188 189Pass the `--png-dir` option pointing at an output directory to compare the output 190images from rendering the "before" and the "after" branches with pdfium_test. 191 192```shell 193$ mkdir ~/output_images 194$ testing/tools/safetynet_compare.py ~/pdf_samples --branch-before before_visual_changes --branch-after after_visual_changes --png-dir ~/output_images 195``` 196 197This will output and automatically open a `~/output_images/compare.html` file 198showing the before/after and the diff. Hover the mouse cursor over the 199before/after image on the left for an easier visual comparison. The "before" 200image is displayed until the cursor hovers over the image, which is then 201replaced with the "after" image. 202 203It is recommended to use `--profiler=none` with this option. 204