1.. _module-pw_fuzzer-guides-using_fuzztest: 2 3======================================== 4pw_fuzzer: Adding Fuzzers Using FuzzTest 5======================================== 6.. pigweed-module-subpage:: 7 :name: pw_fuzzer 8 9.. note:: 10 11 `FuzzTest`_ is currently only supported on Linux and MacOS using Clang. 12 13.. _module-pw_fuzzer-guides-using_fuzztest-toolchain: 14 15---------------------------------------- 16Step 0: Set up FuzzTest for your project 17---------------------------------------- 18.. note:: 19 20 This workflow only needs to be done once for a project. 21 22FuzzTest and its dependencies are not included in Pigweed and need to be added. 23 24See the following: 25 26* :ref:`module-pw_third_party_abseil_cpp-using_upstream` 27* :ref:`module-pw_third_party_fuzztest-using_upstream` 28* :ref:`module-pw_third_party_googletest-using_upstream` 29* :ref:`module-pw_third_party_re2-using_upstream` 30 31.. tab-set:: 32 33 .. tab-item:: GN 34 :sync: gn 35 36 You may not want to use upstream GoogleTest all the time. For example, it 37 may not be supported on your target device. In this case, you can limit it 38 to a specific toolchain used for fuzzing. For example: 39 40 .. code-block:: 41 42 import("$dir_pw_toolchain/host/target_toolchains.gni") 43 44 my_toolchains = { 45 ... 46 clang_fuzz = { 47 name = "my_clang_fuzz" 48 forward_variables_from(pw_toolchain_host.clang_fuzz, "*", ["name"]) 49 pw_unit_test_MAIN = "$dir_pw_fuzzer:fuzztest_main" 50 pw_unit_test_BACKEND = "$dir_pw_fuzzer:gtest" 51 } 52 ... 53 } 54 55 .. tab-item:: CMake 56 :sync: cmake 57 58 FuzzTest is enabled by setting several CMake variables. The easiest way to 59 set these is to extend your ``toolchain.cmake`` file. 60 61 For example: 62 63 .. code-block:: 64 65 include(my_project_toolchain.cmake) 66 67 set(dir_pw_third_party_fuzztest 68 "path/to/fuzztest" 69 CACHE STRING "" FORCE 70 ) 71 set(dir_pw_third_party_googletest 72 "path/to/googletest" 73 CACHE STRING "" FORCE 74 ) 75 set(pw_unit_test_BACKEND 76 "pw_third_party.fuzztest" 77 CACHE STRING "" FORCE 78 ) 79 80 .. tab-item:: Bazel 81 :sync: bazel 82 83 Include Abseil-C++ and GoogleTest in your ``WORKSPACE`` file. For example: 84 85 .. code-block:: 86 87 http_archive( 88 name = "com_google_absl", 89 sha256 = "338420448b140f0dfd1a1ea3c3ce71b3bc172071f24f4d9a57d59b45037da440", 90 strip_prefix = "abseil-cpp-20240116.0", 91 url = "https://github.com/abseil/abseil-cpp/releases/download/20240116.0/abseil-cpp-20240116.0.tar.gz", 92 ) 93 94 git_repository( 95 name = "com_google_googletest", 96 commit = "3b6d48e8d5c1d9b3f9f10ac030a94008bfaf032b", 97 remote = "https://pigweed.googlesource.com/third_party/github/google/googletest", 98 ) 99 100 Then, import the FuzzTest build configurations in your ``.bazelrc`` file 101 by adding and adapting the following: 102 103 .. code-block:: 104 105 # Include FuzzTest build configurations. 106 try-import %workspace%/path/to/pigweed/pw_fuzzer/fuzztest.bazelrc 107 108---------------------------------------- 109Step 1: Write a unit test for the target 110---------------------------------------- 111 112As noted previously, the very first step is to identify one or more target 113behavior that would benefit from testing. See `FuzzTest Use Cases`_ for more 114details on how to identify this code. 115 116Once identified, it is useful to start from a unit test. You may already have a 117unit test writtern, but if not it is likely still be helpful to write one first. 118Many developers are more familiar with writing unit tests, and there are 119detailed guides available. See for example the `GoogleTest documentation`_. 120 121This guide will use code from ``//pw_fuzzer/examples/fuzztest/``. This code 122includes the following object as an example of code that would benefit from 123fuzzing for undefined behavior and from roundtrip fuzzing. 124 125.. note:: 126 127 To keep the example simple, this code uses the standard library. As a result, 128 this code may not work with certain devices. 129 130.. literalinclude:: ../examples/fuzztest/metrics.h 131 :language: cpp 132 :linenos: 133 :start-after: [pwfuzzer_examples_fuzztest-metrics_h] 134 :end-before: [pwfuzzer_examples_fuzztest-metrics_h] 135 136Unit tests for this class might attempt to deserialize previously serialized 137objects and to deserialize invalid data: 138 139.. literalinclude:: ../examples/fuzztest/metrics_unittest.cc 140 :language: cpp 141 :linenos: 142 :start-after: [pwfuzzer_examples_fuzztest-metrics_unittest] 143 :end-before: [pwfuzzer_examples_fuzztest-metrics_unittest] 144 145-------------------------------------------- 146Step 2: Convert your unit test to a function 147-------------------------------------------- 148 149Examine your unit tests and identify any places you have fixed values that could 150vary. Turn your unit test into a function that takes those values as parameters. 151Since fuzzing may not occur on all targets, you should preserve your unit test 152by calling the new function with the previously fixed values. 153 154.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 155 :language: cpp 156 :linenos: 157 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest1] 158 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest1] 159 160.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 161 :language: cpp 162 :linenos: 163 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest3] 164 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest3] 165 166Note that in ``ArbitrarySerializeAndDeserialize`` we no longer assume the 167marshalling will always be successful, and exit early if it is not. You may need 168to make similar modifications to your unit tests if constraints on parameters 169are not expressed by `domains`__ as described below. 170 171.. __: `FuzzTest Domain Reference`_ 172 173-------------------------------------------- 174Step 3: Add a FUZZ_TEST macro invocation 175-------------------------------------------- 176 177Now, include ``"fuzztest/fuzztest.h"`` and pass the test suite name and your 178function name to the ``FUZZ_TEST`` macro. Call ``WithDomains`` on the returned 179object to specify the input domain for each parameter of the function. For 180example: 181 182.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 183 :language: cpp 184 :linenos: 185 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest2] 186 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest2] 187 188.. literalinclude:: ../examples/fuzztest/metrics_fuzztest.cc 189 :language: cpp 190 :linenos: 191 :start-after: [pwfuzzer_examples_fuzztest-metrics_fuzztest4] 192 :end-before: [pwfuzzer_examples_fuzztest-metrics_fuzztest4] 193 194You may know of specific values that are "interesting", i.e. that represent 195boundary conditions, involve, special handling, etc. To guide the fuzzer towards 196these code paths, you can include them as `seeds`_. However, as noted in 197the comments of the examples, it is recommended to include a unit test with the 198original parameters to ensure the code is tested on target devices. 199 200FuzzTest provides more detailed documentation on these topics. For example: 201 202* Refer to `The FUZZ_TEST Macro`_ reference for more details on how to use this 203 macro. 204 205* Refer to the `FuzzTest Domain Reference`_ for details on all the different 206 types of domains supported by FuzzTest and how they can be combined. 207 208* Refer to the `Test Fixtures`_ reference for how to create fuzz tests from unit 209 tests that use GoogleTest fixtures. 210 211------------------------------------ 212Step 4: Add the fuzzer to your build 213------------------------------------ 214Next, indicate that the unit test includes one or more fuzz tests. 215 216.. tab-set:: 217 218 .. tab-item:: GN 219 :sync: gn 220 221 The ``pw_fuzz_test`` template can be used to add the necessary FuzzTest 222 dependency and generate test metadata. 223 224 For example, consider the following ``BUILD.gn``: 225 226 .. literalinclude:: ../examples/fuzztest/BUILD.gn 227 :linenos: 228 :start-after: [pwfuzzer_examples_fuzztest-gn] 229 :end-before: [pwfuzzer_examples_fuzztest-gn] 230 231 .. tab-item:: CMake 232 :sync: cmake 233 234 Unit tests can support fuzz tests by simply adding a dependency on 235 FuzzTest. 236 237 For example, consider the following ``CMakeLists.txt``: 238 239 .. literalinclude:: ../examples/fuzztest/CMakeLists.txt 240 :linenos: 241 :start-after: [pwfuzzer_examples_fuzztest-cmake] 242 :end-before: [pwfuzzer_examples_fuzztest-cmake] 243 244 .. tab-item:: Bazel 245 :sync: bazel 246 247 Unit tests can support fuzz tests by simply adding a dependency on 248 FuzzTest. 249 250 For example, consider the following ``BUILD.bazel``: 251 252 .. literalinclude:: ../examples/fuzztest/BUILD.bazel 253 :linenos: 254 :start-after: [pwfuzzer_examples_fuzztest-bazel] 255 :end-before: [pwfuzzer_examples_fuzztest-bazel] 256 257------------------------ 258Step 5: Build the fuzzer 259------------------------ 260.. tab-set:: 261 262 .. tab-item:: GN 263 :sync: gn 264 265 Build using ``ninja`` on a target that includes your fuzzer with a 266 :ref:`fuzzing toolchain<module-pw_fuzzer-guides-using_fuzztest-toolchain>`. 267 268 Pigweed includes a ``//:fuzzers`` target that builds all tests, including 269 those with fuzzers, using a fuzzing toolchain. You may wish to add a 270 similar top-level to your project. For example: 271 272 .. code-block:: 273 274 group("fuzzers") { 275 deps = [ ":pw_module_tests.run($dir_pigweed/targets/host:host_clang_fuzz)" ] 276 } 277 278 .. tab-item:: CMake 279 :sync: cmake 280 281 Build using ``cmake`` with the FuzzTest and GoogleTest variables set. For 282 example: 283 284 .. code-block:: 285 286 cmake ... \ 287 -Ddir_pw_third_party_fuzztest=path/to/fuzztest \ 288 -Ddir_pw_third_party_googletest=path/to/googletest \ 289 -Dpw_unit_test_BACKEND=pw_third_party.fuzztest 290 291 292 .. tab-item:: Bazel 293 :sync: bazel 294 295 By default, ``bazel`` will simply omit the fuzz tests and build unit 296 tests. To build these tests as fuzz tests, specify the ``fuzztest`` 297 config. For example: 298 299 .. code-block:: sh 300 301 bazel build //... --config=fuzztest 302 303---------------------------------- 304Step 6: Running the fuzzer locally 305---------------------------------- 306.. TODO: b/281138993 - Add tooling to make it easier to find and run fuzzers. 307 308.. tab-set:: 309 310 .. tab-item:: GN 311 :sync: gn 312 313 When building. Most toolchains will simply omit the fuzz tests and build 314 and run unit tests. A 315 :ref:`fuzzing toolchain<module-pw_fuzzer-guides-using_fuzztest-toolchain>` 316 will include the fuzzers, but only run them for a limited time. This makes 317 them suitable for automated testing as in CQ. 318 319 If you used the top-level ``//:fuzzers`` described in the previous 320 section, you can find available fuzzers using the generated JSON test 321 metadata file: 322 323 .. code-block:: sh 324 325 jq '.[] | select(contains({tags: ["fuzztest"]}))' \ 326 out/host_clang_fuzz/obj/pw_module_tests.testinfo.json 327 328 To run a fuzz with different options, you can pass additional flags to the 329 fuzzer binary. This binary will be in a subdirectory related to the 330 toolchain. For example: 331 332 .. code-block:: sh 333 334 out/host_clang_fuzz/obj/my_module/test/metrics_test \ 335 --fuzz=MetricsTest.Roundtrip 336 337 Additional `sanitizer flags`_ may be passed uisng environment variables. 338 339 .. tab-item:: CMake 340 :sync: cmake 341 342 When built with FuzzTest and GoogleTest, the fuzzer binaries can be run 343 directly from the CMake build directory. By default, the fuzzers will only 344 run for a limited time. This makes them suitable for automated testing as 345 in CQ. To run a fuzz with different options, you can pass additional flags 346 to the fuzzer binary. 347 348 For example: 349 350 .. code-block:: sh 351 352 build/my_module/metrics_test --fuzz=MetricsTest.Roundtrip 353 354 .. tab-item:: Bazel 355 :sync: bazel 356 357 By default, ``bazel`` will simply omit the fuzz tests and build and run 358 unit tests. To build these tests as fuzz tests, specify the "fuzztest" 359 config. For example: 360 361 .. code-block:: sh 362 363 bazel test //... --config=fuzztest 364 365 This will build the tests as fuzz tests, but only run them for a limited 366 time. This makes them suitable for automated testing as in CQ. 367 368 To run a fuzz with different options, you can use ``run`` and pass 369 additional flags to the fuzzer binary. For example: 370 371 .. code-block:: sh 372 373 bazel run //my_module:metrics_test --config=fuzztest \ 374 --fuzz=MetricsTest.Roundtrip 375 376Running the fuzzer should produce output similar to the following: 377 378.. code-block:: 379 380 [.] Sanitizer coverage enabled. Counter map size: 21290, Cmp map size: 262144 381 Note: Google Test filter = MetricsTest.Roundtrip 382 [==========] Running 1 test from 1 test suite. 383 [----------] Global test environment set-up. 384 [----------] 1 test from MetricsTest 385 [ RUN ] MetricsTest.Roundtrip 386 [*] Corpus size: 1 | Edges covered: 131 | Fuzzing time: 504.798us | Total runs: 1.00e+00 | Runs/secs: 1 387 [*] Corpus size: 2 | Edges covered: 133 | Fuzzing time: 934.176us | Total runs: 3.00e+00 | Runs/secs: 3 388 [*] Corpus size: 3 | Edges covered: 134 | Fuzzing time: 2.384383ms | Total runs: 5.30e+01 | Runs/secs: 53 389 [*] Corpus size: 4 | Edges covered: 137 | Fuzzing time: 2.732274ms | Total runs: 5.40e+01 | Runs/secs: 54 390 [*] Corpus size: 5 | Edges covered: 137 | Fuzzing time: 7.275553ms | Total runs: 2.48e+02 | Runs/secs: 248 391 392.. TODO: b/282560789 - Add guides/improve_fuzzers.rst 393.. TODO: b/281139237 - Add guides/continuous_fuzzing.rst 394.. ---------- 395.. Next steps 396.. ---------- 397.. Once you have a working fuzzer, the next steps are to: 398 399.. * `Run it continuously on a fuzzing infrastructure <continuous_fuzzing>`_. 400.. * `Measure its code coverage and improve it <improve_fuzzers>`_. 401 402.. _FuzzTest: https://github.com/google/fuzztest 403.. _FuzzTest Domain Reference: https://github.com/google/fuzztest/blob/main/doc/domains-reference.md 404.. _FuzzTest Use Cases: https://github.com/google/fuzztest/blob/main/doc/use-cases.md 405.. _GoogleTest documentation: https://google.github.io/googletest/ 406.. _Test Fixtures: https://github.com/google/fuzztest/blob/main/doc/fixtures.md 407.. _The FUZZ_TEST Macro: https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md 408.. _sanitizer flags: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags 409.. _seeds: https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#initial-seeds 410