1.. _docs-automated-analysis: 2 3================== 4Automated analysis 5================== 6 7The correctness and style of Pigweed's source code is continuously verified 8using a suite of automated tools. We also make it easy to use the same tools 9to verify the code of projects using Pigweed. 10 11------- 12Summary 13------- 14On presubmit or in CI we verify Pigweed using: 15 16* pylint 17* mypy 18* clang-tidy 19* AddressSanitizer (asan) 20* ThreadSanitizer (tsan) 21* UndefinedBehaviorSanitizer (ubsan) 22* OSS-Fuzz 23 24The rest of this document discusses these tools and their configuration in 25greater detail, and how to use them in your own project. 26 27-------------- 28Analysis tools 29-------------- 30 31Static analysis 32=============== 33 34PyLint 35------ 36`PyLint`_ is a customizable Python linter. Pigweed complies with almost all 37the default checks; see `.pylintrc`_ for details. PyLint detects problems such 38as overly broad catch statements, unused arguments/variables, and mutable 39default parameter values. 40 41For upstream Pigweed, PyLint can be run with ``ninja python.lint.pylint`` or 42``ninja python.lint``. It's also included in a variety of presubmit steps, 43like ``static_analysis`` and ``python_checks.gn_python_check``. See the 44`Enabling analysis for your project`_ section to learn how to run PyLint on 45your Pigweed-based project. 46 47.. _PyLint: https://pylint.org/ 48.. _.pylintrc: https://cs.pigweed.dev/pigweed/+/main:.pylintrc 49 50Mypy 51---- 52Python 3 allows for `type annotations`_ for variables, function arguments, and 53return values. Most, but not all, of Pigweed's Python code has type 54annotations, and these annotations have caught real bugs in code that didn't 55yet have unit tests. `Mypy`_ is an analysis tool that enforces these 56annotations. 57 58Mypy helps find bugs like when a string is passed into a function that expects 59a list of strings---since both are iterables this bug might otherwise be hard 60to track down. 61 62Mypy can be run with ``ninja python.lint.mypy`` or ``ninja python.lint``. It's 63also included in a variety of presubmit steps, like ``static_analysis`` and 64``python_checks.gn_python_check``. 65 66.. _type annotations: https://docs.python.org/3/library/typing.html 67.. _Mypy: http://mypy-lang.org/ 68 69clang-tidy 70---------- 71`clang-tidy`_ is a C++ "linter" and static analysis tool. It identifies 72bug-prone patterns (e.g., use after move), non-idiomatic usage (e.g., creating 73``std::unique_ptr`` with ``new`` rather than ``std::make_unique``), and 74performance issues (e.g., unnecessary copies of loop variables). 75 76While powerful, clang-tidy defines a very large number of checks, many of which 77are special-purpose (e.g., only applicable to FPGA HLS code, or code using the 78`Abseil`_ library) or have high false positive rates. Pigweed enables over 50 79checks which are relevant to an embedded C/C++ library and have good 80signal-to-noise ratios. The full list of Pigweed's checks is in `.clang-tidy`_. 81 82We do not currently enable the `Clang Static Analyzers`_ because they suffer 83from false positives, and their findings are time-consuming to manually verify. 84 85clang-tidy can be run with ``ninja static_analysis`` or ``pw presubmit --step 86static_analysis``. Note that as a static analysis tool, clang-tidy will not 87produce any runnable binaries: it simply analyzes the source files. 88 89.. _clang-tidy: https://clang.llvm.org/extra/clang-tidy/ 90.. _Abseil: https://abseil.io/ 91.. _.clang-tidy: https://cs.pigweed.dev/pigweed/+/main:.clang-tidy 92.. _Clang Static Analyzers: https://clang-analyzer.llvm.org/available_checks.html 93 94 95Clang sanitizers 96================ 97We run all of Pigweed's unit tests with the additional instrumentation 98described in this section. For more detail about these sanitizers, see the 99`Github documentation`_. 100 101* asan: `AddressSanitizer`_ detects memory errors such as out-of-bounds access 102 and use-after-free. 103* tsan: `ThreadSanitizer`_ detects data races. 104* ubsan: `UndefinedBehaviorSanitizer`_ is a fast undefined behavior detector. 105 We use the default ``-fsanitize=undefined`` option. 106 107.. note:: 108 Pigweed does not currently support `MemorySanitizer`_ (msan). See 109 :bug:`234876100` for details. 110 111The exact configurations we use for these sanitizers are in 112`pw_toolchain/host_clang/BUILD.gn <https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/host_clang/BUILD.gn>`_. 113You can see the current status of the sanitizer builds in the `Pigweed CI 114console`_, as ``pigweed-linux-san-*``. 115 116Unlike clang-tidy, the clang sanitizers are runtime instrumentation: the 117instrumented binary needs to be run for issues to be detected. 118 119.. _Github documentation: https://github.com/google/sanitizers 120.. _AddressSanitizer: https://clang.llvm.org/docs/AddressSanitizer.html 121.. _MemorySanitizer: https://clang.llvm.org/docs/MemorySanitizer.html 122.. _Pigweed CI console: https://ci.chromium.org/p/pigweed/g/pigweed/console 123.. _ThreadSanitizer: https://clang.llvm.org/docs/ThreadSanitizer.html 124.. _UndefinedBehaviorSanitizer: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html 125 126 127Fuzzers 128======= 129`Fuzz testing`_ detects errors in software by providing it with randomly 130generated inputs. We use `OSS-fuzz`_ to continuously uncover potential 131vulnerabilities in Pigweed. `Dashboard with Pigweed's latest results`_. See 132the :ref:`module-pw_fuzzer` module documentation for more details. 133 134.. _Dashboard with Pigweed's latest results: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#pigweed 135.. _Fuzz testing: https://en.wikipedia.org/wiki/Fuzzing 136.. _OSS-fuzz: https://github.com/google/oss-fuzz 137 138.. _Enabling analysis for your project: 139 140---------------------------------- 141Enabling analysis for your project 142---------------------------------- 143 144GN 145== 146 147PyLint and Mypy 148--------------- 149PyLint and Mypy can be configured to run every time your project is built by 150adding ``python.lint`` to your default build group. (You can also add one or both 151individually using ``python.lint.mypy`` and ``python.lint.pylint``.) Likewise, 152these can be added to individual presubmit steps (`examples`_). You can also 153directly include the `python_checks.gn_python_lint`_ presubmit step. 154 155.. _examples: https://cs.opensource.google/search?q=file:pigweed_presubmit.py%20%22python.lint%22&sq=&ss=pigweed%2Fpigweed 156.. _python_checks.gn_python_lint: https://cs.pigweed.dev/pigweed/+/main:pw_presubmit/py/pw_presubmit/python_checks.py?q=file:python_checks.py%20gn_python_lint&ss=pigweed%2Fpigweed 157 158clang-tidy 159---------- 160`pw_toolchain/static_analysis_toolchain.gni`_ provides the 161``pw_static_analysis_toolchain`` template that can be used to create a build 162group performing static analysis. See :ref:`module-pw_toolchain` documentation 163for more details. This group can then be added as a presubmit step using 164pw_presubmit. 165 166You can place a ``.clang-tidy`` file at the root of your repository to control 167which checks are executed. See the `clang documentation`_ for a discussion of how 168the tool chooses which ``.clang-tidy`` files to apply when run on a particular 169source file. 170 171.. _pw_toolchain/static_analysis_toolchain.gni: https://cs.pigweed.dev/pigweed/+/main:pw_toolchain/static_analysis_toolchain.gni 172.. _clang documentation: https://clang.llvm.org/extra/clang-tidy/ 173 174Clang sanitizers 175---------------- 176There are two ways to enable sanitizers for your build. 177 178GN args on debug toolchains 179^^^^^^^^^^^^^^^^^^^^^^^^^^^ 180If you are already building your tests with one of the following toolchains (or 181a toolchain derived from one of them): 182 183* ``pw_toolchain_host_clang.debug`` 184* ``pw_toolchain_host_clang.speed_optimized`` 185* ``pw_toolchain_host_clang.size_optimized`` 186 187you can enable the clang sanitizers simply by setting the gn arg 188``pw_toolchain_SANITIZERS`` to the desired subset of 189``["address", "thread", "undefined"]``. 190 191Example 192....... 193If your project defines a toolchain ``host_clang_debug`` that is derived from 194one of the above toolchains, and you'd like to run the ``pw_executable`` target 195``sample_binary`` defined in the ``BUILD.gn`` file in ``examples/sample`` with 196asan, you would run, 197 198.. code-block:: bash 199 200 gn gen out --args='pw_toolchain_SANITIZERS=["address"]' 201 ninja -C out host_clang_debug/obj/example/sample/bin/sample_binary 202 out/host_clang_debug/obj/example/sample/bin/sample_binary 203 204Sanitizer toolchains 205^^^^^^^^^^^^^^^^^^^^ 206Otherwise, instead of using ``gn args`` you can build your tests with the 207appropriate toolchain from the following list (or a toolchain derived from one 208of them): 209 210* ``pw_toolchain_host_clang.asan`` 211* ``pw_toolchain_host_clang.ubsan`` 212* ``pw_toolchain_host_clang.tsan`` 213 214See the :ref:`module-pw_toolchain` module documentation for more 215about Pigweed toolchains. 216 217Bazel 218===== 219 220.. _docs-automated-analysis-clang-sanitizers: 221 222Clang sanitizers 223---------------- 224If you're using Pigweed's own host toolchain configuration, you can enable 225AddressSanitizer by building with the appropriate flag: 226 227.. code-block:: sh 228 229 bazelisk build --@pigweed//pw_toolchain/host_clang:asan //... 230 231If you're building your own toolchain, you can add 232``@pigweed//pw_toolchain_bazel/flag_sets:asan`` to it. 233 234Fuzzers 235======= 236See the :ref:`module-pw_fuzzer` module documentation. 237