1# Reducing Variance 2 3<a name="disabling-cpu-frequency-scaling" /> 4 5## Disabling CPU Frequency Scaling 6 7If you see this error: 8 9``` 10***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. 11``` 12 13you might want to disable the CPU frequency scaling while running the 14benchmark, as well as consider other ways to stabilize the performance of 15your system while benchmarking. 16 17Exactly how to do this depends on the Linux distribution, 18desktop environment, and installed programs. Specific details are a moving 19target, so we will not attempt to exhaustively document them here. 20 21One simple option is to use the `cpupower` program to change the 22performance governor to "performance". This tool is maintained along with 23the Linux kernel and provided by your distribution. 24 25It must be run as root, like this: 26 27```bash 28sudo cpupower frequency-set --governor performance 29``` 30 31After this you can verify that all CPUs are using the performance governor 32by running this command: 33 34```bash 35cpupower frequency-info -o proc 36``` 37 38The benchmarks you subsequently run will have less variance. 39 40<a name="reducing-variance" /> 41 42## Reducing Variance in Benchmarks 43 44The Linux CPU frequency governor [discussed 45above](user_guide#disabling-cpu-frequency-scaling) is not the only source 46of noise in benchmarks. Some, but not all, of the sources of variance 47include: 48 491. On multi-core machines not all CPUs/CPU cores/CPU threads run the same 50 speed, so running a benchmark one time and then again may give a 51 different result depending on which CPU it ran on. 522. CPU scaling features that run on the CPU, like Intel's Turbo Boost and 53 AMD Turbo Core and Precision Boost, can temporarily change the CPU 54 frequency even when the using the "performance" governor on Linux. 553. Context switching between CPUs, or scheduling competition on the CPU the 56 benchmark is running on. 574. Intel Hyperthreading or AMD SMT causing the same issue as above. 585. Cache effects caused by code running on other CPUs. 596. Non-uniform memory architectures (NUMA). 60 61These can cause variance in benchmarks results within a single run 62(`--benchmark_repetitions=N`) or across multiple runs of the benchmark 63program. 64 65Reducing sources of variance is OS and architecture dependent, which is one 66reason some companies maintain machines dedicated to performance testing. 67 68Some of the easier and effective ways of reducing variance on a typical 69Linux workstation are: 70 711. Use the performance governor as [discussed 72above](user_guide#disabling-cpu-frequency-scaling). 731. Disable processor boosting by: 74 ```sh 75 echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost 76 ``` 77 See the Linux kernel's 78 [boost.txt](https://www.kernel.org/doc/Documentation/cpu-freq/boost.txt) 79 for more information. 802. Set the benchmark program's task affinity to a fixed cpu. For example: 81 ```sh 82 taskset -c 0 ./mybenchmark 83 ``` 843. Disabling Hyperthreading/SMT. This can be done in the Bios or using the 85 `/sys` file system (see the LLVM project's [Benchmarking 86 tips](https://llvm.org/docs/Benchmarking.html)). 874. Close other programs that do non-trivial things based on timers, such as 88 your web browser, desktop environment, etc. 895. Reduce the working set of your benchmark to fit within the L1 cache, but 90 do be aware that this may lead you to optimize for an unrealistic 91 situation. 92 93Further resources on this topic: 94 951. The LLVM project's [Benchmarking 96 tips](https://llvm.org/docs/Benchmarking.html). 971. The Arch Wiki [Cpu frequency 98scaling](https://wiki.archlinux.org/title/CPU_frequency_scaling) page. 99