1.. _module-pw_perf_test: 2 3============ 4pw_perf_test 5============ 6 7.. pigweed-module:: 8 :name: pw_perf_test 9 10 - **Simple**: Automatically manages boilerplate like iterations and durations. 11 - **Easy**: Uses an intuitive API that resembles GoogleTest. 12 - **Reusable**: Integrates with modules like ``pw_log`` that you already use. 13 14Pigweed's perf test module provides an easy way to measure performance on 15any test setup! 16 17--------------- 18Getting started 19--------------- 20You can add a simple performance test using the follow steps: 21 22Configure your toolchain 23======================== 24If necessary, configure your toolchain for performance testing: 25 26.. note:: Currently, ``pw_perf_test`` provides build integration with Bazel and 27 GN. Performance tests can be built in CMake, but must be built as regular 28 executables. 29 30.. tab-set:: 31 32 .. tab-item:: Bazel 33 :sync: bazel 34 35 - ``pw_perf_test_timer_backend``: Sets the backend used to measure 36 durations. Options include: 37 38 - ``@pigweed//pw_perf_test:chrono_timer``: Uses 39 ``pw_chrono::SystemClock`` to measure time. 40 - ``@pigweed//pw_perf_test:arm_cortex_timer``: Uses cycle count 41 registers available on ARM-Cortex to measure time. 42 43 - Currently, only the logging event handler is supported for Bazel. 44 45 .. tab-item:: GN 46 :sync: gn 47 48 - ``pw_perf_test_TIMER_INTERFACE_BACKEND``: Sets the backend used to 49 measure durations. Options include: 50 51 - ``"$dir_pw_perf_test:chrono_timer"``: Uses 52 ``pw_chrono::SystemClock`` to measure time. 53 - ``"$dir_pw_perf_test:arm_cortex_timer"``: Uses cycle count 54 registers available on ARM-Cortex to measure time. 55 56 - ``pw_perf_test_MAIN_FUNCTION``: Indicates the GN target that provides 57 a ``main`` function that sets the event handler and runs tests. The 58 default is ``"$dir_pw_perf_test:logging_main"``. 59 60Write a test function 61===================== 62Write a test function that exercises the behavior you wish to benchmark. For 63this example, we will simulate doing work with: 64 65.. literalinclude:: examples/example_perf_test.cc 66 :language: cpp 67 :linenos: 68 :start-after: [pw_perf_test_examples-simulate_work] 69 :end-before: [pw_perf_test_examples-simulate_work] 70 71Creating a performance test is as simple as using the ``PW_PERF_TEST_SIMPLE`` 72macro to name the function and optionally provide arguments to it: 73 74.. literalinclude:: examples/example_perf_test.cc 75 :language: cpp 76 :linenos: 77 :start-after: [pw_perf_test_examples-simple_example] 78 :end-before: [pw_perf_test_examples-simple_example] 79 80If you need to do additional setup as part of your test, you can use the 81``PW_PERF_TEST`` macro, which provides an explicit ``pw::perf_test::State`` 82reference. the behavior to be benchmarked should be put in a loop that checks 83``State::KeepRunning()``: 84 85.. literalinclude:: examples/example_perf_test.cc 86 :language: cpp 87 :linenos: 88 :start-after: [pw_perf_test_examples-full_example] 89 :end-before: [pw_perf_test_examples-full_example] 90 91You can even use lambdas in place of standalone functions: 92 93.. literalinclude:: examples/example_perf_test.cc 94 :language: cpp 95 :linenos: 96 :start-after: [pw_perf_test_examples-lambda_example] 97 :end-before: [pw_perf_test_examples-lambda_example] 98 99.. _module-pw_perf_test-pw_perf_test: 100 101Build Your Test 102=============== 103.. tab-set:: 104 105 .. tab-item:: Bazel 106 :sync: bazel 107 108 Add your performance test to the build using the ``pw_cc_perf_test`` 109 rule from ``//pw_perf_test:pw_cc_perf_test.bzl``. 110 111 **Arguments** 112 113 * All ``native.cc_binary`` arguments are supported. 114 115 **Example** 116 117 .. code-block:: 118 119 load("//pw_perf_test:pw_cc_perf_test.bzl", "pw_cc_perf_test") 120 121 pw_cc_perf_test( 122 name = "my_perf_test", 123 srcs = ["my_perf_test.cc"], 124 ) 125 126 .. tab-item:: GN 127 :sync: gn 128 129 Add your performance test to the build using the ``pw_perf_test`` 130 template. This template creates two sub-targets. 131 132 * ``<target_name>.lib``: The test sources without a main function. 133 * ``<target_name>``: The test suite binary, linked against 134 ``pw_perf_test_MAIN_FUNCTION``. 135 136 **Arguments** 137 138 * All ``pw_executable`` arguments are supported. 139 * ``enable_if``: Boolean indicating whether the test should be built. If 140 false, replaces the test with an empty target. Defaults to true. 141 142 **Example** 143 144 .. code-block:: 145 146 import("$dir_pw_perf_test/perf_test.gni") 147 148 pw_perf_test("my_perf_test") { 149 sources = [ "my_perf_test.cc" ] 150 enable_if = device_has_1m_flash 151 } 152 153Run your test 154============= 155.. tab-set:: 156 157 .. tab-item:: GN 158 :sync: gn 159 160 To run perf tests from GN, locate the associated binaries from the ``out`` 161 directory and run/flash them manually. 162 163 .. tab-item:: Bazel 164 :sync: bazel 165 166 Use the default Bazel run command: ``bazel run //path/to:target``. 167 168------------- 169API reference 170------------- 171 172Macros 173====== 174 175.. doxygendefine:: PW_PERF_TEST 176 177.. doxygendefine:: PW_PERF_TEST_SIMPLE 178 179EventHandler 180============ 181 182.. doxygenclass:: pw::perf_test::EventHandler 183 :members: 184 185------ 186Design 187------ 188 189``pw_perf_test`` uses a ``Framework`` singleton similar to that of 190``pw_unit_test``. This singleton is statically created, and tests declared using 191macros such as ``PW_PERF_TEST`` will automatically register themselves with it. 192 193A provided ``main`` function interacts with the ``Framework`` by calling 194``pw::perf_test::RunAllTests`` and providing an ``EventHandler``. For each 195registered test, the ``Framework`` creates a ``State`` object and passes it to 196the test function. 197 198The State object tracks the number of iterations. It expects the test function 199to include a loop with the condition of ``State::KeepRunning``. This loop 200should include the behavior being banchmarked, e.g. 201 202.. code-block:: cpp 203 204 while (state.KeepRunning()) { 205 // Code to be benchmarked. 206 } 207 208In particular, ``State::KeepRunning`` should be called exactly once before the 209first iteration, as in a ``for`` or ``while`` loop. The ``State`` object will 210use the timer facade to measure the elapsed duration between successive calls to 211``State::KeepRunning``. 212 213Additionally, the ``State`` object receives a reference to the ``EventHandler`` 214from the ``Framework``, and uses this to report both test progress and 215performance measurements. 216 217Timers 218====== 219Currently, Pigweed provides two implementations of the timer interface. 220Consumers may provide additional implementations and use them as a backend for 221the timer facade. 222 223Chrono Timer 224------------ 225This timer depends :ref:`module-pw_chrono` and will only measure performance in 226terms of nanoseconds. It is the default for performance tests on host. 227 228Cycle Count Timer 229----------------- 230On ARM Cortex devices, clock cycles may more accurately measure the actual 231performance of a benchmark. 232 233This implementation is OS-agnostic, as it directly accesses CPU registers. 234It enables the `DWT register`_ through the `DEMCR register`_. While this 235provides cycle counts directly from the CPU, it notably overflows if the 236duration of a test exceeding 2^32 clock cycles. At 100 MHz, this is 237approximately 43 seconds. 238 239.. warning:: 240 The interface only measures raw clock cycles and does not take into account 241 other possible sources of pollution such as LSUs, Sleeps and other registers. 242 `Read more on the DWT methods of counting instructions.`__ 243 244.. __: `DWT methods`_ 245 246EventHandlers 247============= 248Currently, Pigweed provides one implementation of ``EventHandler``. Consumers 249may provide additional implementations and use them by providing a dedicated 250``main`` function that passes the handler to ``pw::perf_test::RunAllTests``. 251 252LoggingEventHandler 253------------------- 254The default method of running performance tests uses a ``LoggingEventHandler``. 255This event handler only logs the test results to the console and nothing more. 256It was chosen as the default method due to its portability and to cut down on 257the time it would take to implement other printing log handlers. Make sure to 258set a ``pw_log`` backend. 259 260------- 261Roadmap 262------- 263- `CMake support <https://g-issues.pigweed.dev/issues/309637691>`_ 264- `Unified framework <https://g-issues.pigweed.dev/issues/309639171>`_. 265 266.. _DWT register: https://developer.arm.com/documentation/ddi0337/e/System-Debug/DWT?lang=en 267.. _DEMCR register: https://developer.arm.com/documentation/ddi0337/e/CEGHJDCF 268.. _DWT methods: https://developer.arm.com/documentation/ka001499/1-0/ 269