1.. _docs-get-started-upstream: 2 3============================================= 4Get started with upstream Pigweed development 5============================================= 6This guide will walk you through the typical upstream development workflow. 7"Upstream development" means that you're contributing to the main Pigweed 8repository, ``https://pigweed.googlesource.com/pigweed/pigweed``. See 9:ref:`docs-get-started` if you're looking for instructions on how to use 10Pigweed in your own project. 11 12.. _prerequisites: 13 14Prerequisites 15============= 16If you haven't already, :ref:`docs-first-time-setup`. 17 18------------- 19Express setup 20------------- 21If you'd like to skip the detailed explanations, below is the shorter version 22of getting setup for Pigweed. If you run into trouble, ensure you've followed 23all the steps to :ref:`docs-first-time-setup`. The express setup 24configures Pigweed's watcher for three targets to give a taste of Pigweed: 25 26#. **Host** - Mac, Linux, or Windows. Builds and runs tests 27#. **Device/STM32F429** - Build only; Optionally, the STM32F429I-DISC1 kit to 28 follow along later in the guide to run tests directly on said device(s) 29#. **Docs** - Builds the Pigweed docs 30 31To get setup: 32 33#. Make sure you have Git and Python installed and on your path. 34 35#. Clone Pigweed and bootstrap the environment (compiler setup & more). **Be 36 patient, this step downloads ~1GB of LLVM, GCC, and other tooling**. 37 38 .. code-block:: bash 39 40 $ cd ~ 41 $ git clone https://pigweed.googlesource.com/pigweed/pigweed 42 ... 43 $ cd pigweed 44 $ source ./bootstrap.sh (On Linux & Mac) 45 $ bootstrap.bat (On Windows) 46 ... 47 48 .. tip:: 49 50 If you use the `Fish shell <https://fishshell.com/>`_ run `source 51 ./bootstrap.fish` instead. 52 53#. Configure the GN build. 54 55 .. code-block:: bash 56 57 $ gn gen out 58 Done. Made 1047 targets from 91 files in 114ms 59 60#. Start the watcher. The watcher will invoke Ninja to build all the targets 61 62 .. code-block:: bash 63 64 $ pw watch 65 66 ▒█████▄ █▓ ▄███▒ ▒█ ▒█ ░▓████▒ ░▓████▒ ▒▓████▄ 67 ▒█░ █░ ░█▒ ██▒ ▀█▒ ▒█░ █ ▒█ ▒█ ▀ ▒█ ▀ ▒█ ▀█▌ 68 ▒█▄▄▄█░ ░█▒ █▓░ ▄▄░ ▒█░ █ ▒█ ▒███ ▒███ ░█ █▌ 69 ▒█▀ ░█░ ▓█ █▓ ░█░ █ ▒█ ▒█ ▄ ▒█ ▄ ░█ ▄█▌ 70 ▒█ ░█░ ░▓███▀ ▒█▓▀▓█░ ░▓████▒ ░▓████▒ ▒▓████▀ 71 72 20200707 17:24:06 INF Starting Pigweed build watcher 73 20200707 17:24:06 INF Will build [1/1]: out 74 20200707 17:24:06 INF Attaching filesystem watcher to $HOME/wrk/pigweed/... 75 20200707 17:24:06 INF Triggering initial build... 76 ... 77 78#. **Congratulations, you're ready to go!** Now take Pigweed for a spin by 79 making a test fail. 80 81#. With the watcher running in a separate window, edit 82 ``pw_status/status_test.cc`` to make an expectation fail; for example, add 83 ``EXPECT_EQ(0, 1);`` in a test. 84 85#. Save the file. Observe the watcher rebuild & retest, and fail. Restore the 86 test if you feel like it. 87 88#. Open the generated docs in ``out/docs/gen/docs/html/index.html`` in your 89 browser. 90 91#. Edit ``docs/getting_started.rst`` (this file!) and make any change. Save. 92 See the watcher rebuild the docs. Reload your browser, and see the changes. 93 94See below for equivalent Windows commands, and for more details on what each 95part does. 96 97**Note:** After running bootstrap once, use ``source ./activate.sh`` (or 98``activate.bat`` on Windows) to re-activate the environment without 99re-bootstrapping. 100 101--------- 102Bootstrap 103--------- 104Once you satisfied the prerequisites, you will be able to clone Pigweed and 105run the bootstrap that initializes the Pigweed virtual environment. The 106bootstrap may take several minutes to complete, so please be patient. 107 108**Linux & macOS** 109 110.. code-block:: bash 111 112 $ git clone https://pigweed.googlesource.com/pigweed/pigweed pigweed 113 $ cd pigweed 114 $ source ./bootstrap.sh 115 116**Windows** 117 118.. code-block:: batch 119 120 :: Run git commands from the shell you set up to use with Git during install. 121 > git clone https://pigweed.googlesource.com/pigweed/pigweed %HOMEPATH%\pigweed 122 > cd %HOMEPATH%\pigweed 123 > bootstrap.bat 124 125Below is a real-time demo with roughly what you should expect to see as output: 126 127.. image:: https://storage.googleapis.com/pigweed-media/pw_env_setup_demo.gif 128 :width: 800 129 :alt: build example using pw watch 130 131Congratulations, you are now set up to start using Pigweed! 132 133.. _activate-pigweed-environment: 134 135--------------------------------- 136Activate your Pigweed environment 137--------------------------------- 138After going through the initial setup process, your current terminal will be in 139the Pigweed development environment that provides all the tools you should need 140to develop on Pigweed. If you leave that session, you can activate the 141environment in a new session with the following command: 142 143**Linux & macOS** 144 145.. code-block:: bash 146 147 $ source ./activate.sh 148 149**Windows** 150 151.. code-block:: batch 152 153 > activate.bat 154 155Some major changes may require triggering the bootstrap again, so if you run 156into host tooling changes after a pull it may be worth re-running bootstrap. 157 158---------------------- 159Build Pigweed for host 160---------------------- 161Pigweed's primary build system is GN/Ninja based. There are CMake and Bazel 162builds in-development, but they are incomplete and don't have feature parity 163with the GN build. We strongly recommend you stick to the GN build system. 164 165GN (Generate Ninja) just does what it says on the tin; GN generates 166`Ninja <https://ninja-build.org/>`_ build files. 167 168The default GN configuration generates build files that allow you to build host 169binaries, device binaries, and upstream documentation all in one Ninja 170invocation. 171 172Run GN as seen below: 173 174.. code-block:: bash 175 176 $ gn gen out 177 178.. note:: 179 ``out`` is simply the directory the build files are saved to. Unless 180 this directory is deleted or you desire to do a clean build, there's no need 181 to run GN again; just rebuild using Ninja directly. 182 183.. warning:: 184 Unless your build directory (the ``out`` in ``gn gen out``) is exactly one 185 directory away from the project root directory (the Pigweed repo root in this 186 case), there will be issues finding source files while debugging and while 187 generating coverage reports. This is due an issue in upstream LLVM reordering 188 debug and coverage path mappings (tracked by 189 `b/278898014 <https://issuetracker.google.com/278898014>`_ and 190 `b/278906020 <https://issuetracker.google.com/278906020>`_). **Stick to 191 simple, single directory build directories for now.** 192 193Now that we have build files, it's time to build Pigweed! 194 195Now you *could* manually invoke the host build using ``ninja -C out`` every 196time you make a change, but that's tedious. Instead, let's use ``pw_watch``. 197 198Go ahead and start ``pw_watch``: 199 200.. code-block:: bash 201 202 $ pw watch 203 204When ``pw_watch`` starts up, it will automatically build the directory we 205generated in ``out``. Additionally, ``pw_watch`` watches source code files for 206changes, and triggers a Ninja build whenever it notices a file has been saved. 207You might be surprised how much time it can save you! 208 209With ``pw watch`` running, try modifying 210``pw_status/public/pw_status/status.h`` and watch the build re-trigger when you 211save the file. 212 213See below for a demo of this in action: 214 215.. image:: https://storage.googleapis.com/pigweed-media/pw_watch_build_demo.gif 216 :width: 800 217 :alt: build example using pw watch 218 219------------------ 220Running unit tests 221------------------ 222Fun fact, you've been running the unit tests already! Ninja builds targeting 223the host automatically build and run the unit tests. Unit tests err on the side 224of being quiet in the success case, and only output test results when there's a 225failure. 226 227To see a test failure, modify ``pw_status/status_test.cc`` to fail by changing 228one of the strings in the "KnownString" test. 229 230.. image:: https://storage.googleapis.com/pigweed-media/pw_watch_test_demo.gif 231 :width: 800 232 :alt: example test failure using pw watch 233 234Running tests as part of the build isn't particularly expensive because GN 235caches passing tests. Each time you build, only the tests that are affected 236(whether directly or transitively) by the code changes since the last build 237will be re-built and re-run. 238 239Try running the ``pw_status`` test manually: 240 241.. code-block:: bash 242 243 $ ./out/pw_strict_host_{clang,gcc}_debug/obj/pw_status/test/status_test 244 245Depending on your host OS, the compiler will default to either ``clang`` or 246``gcc``. 247 248--------------------- 249Building for a device 250--------------------- 251A Pigweed "target" is a build configuration that includes a toolchain, default 252library configurations, and more to result in binaries that run natively on the 253target. With the default build invocation, you're already building for a device 254target (the STMicroelectronics STM32F429I-DISC1) in parallel with the host 255build! 256 257If you want to build JUST for the device, you can kick of watch with: 258 259.. code-block:: bash 260 261 $ pw watch stm32f429i 262 263This is equivalent to the following Ninja invocation: 264 265.. code-block:: bash 266 267 $ ninja -C out stm32f429i 268 269------------------------- 270Running tests on a device 271------------------------- 272While tests run automatically on the host, it takes a few more steps to get 273tests to run automatically on a device, too. Even though we've verified tests 274pass on the host, it's crucial to verify the same with on-device testing. We've 275encountered some unexpected bugs that can only be found by running the unit 276tests directly on the device. 277 2781. Connect device(s) 279==================== 280Connect any number of STM32F429I-DISC1 boards to your computer using the mini 281USB port on the board (**not** the micro USB). Pigweed will automatically 282detect the boards and distribute the tests across the devices. More boards = 283faster tests! Keep in mind that you may have to make some environment specific 284updates to ensure you have permissions to use the USB device. For example, on 285Linux you may need to update your udev rules and ensure you're in the plugdev 286and dialout groups. 287 288.. image:: https://storage.googleapis.com/pigweed-media/stm32f429i-disc1_connected.jpg 289 :width: 800 290 :alt: development boards connected via USB 291 2922. Launch test server 293===================== 294To allow Ninja to run tests on an arbitrary number of devices, Ninja will send 295test requests to a server running in the background. Launch the server in 296another window using the command below (remember, you'll need to activate the 297Pigweed environment first). 298 299.. code-block:: bash 300 301 $ stm32f429i_disc1_test_server 302 303**Note:** If you attach or detach any more boards to your workstation you'll 304need to relaunch this server. 305 3063. Configure GN 307=============== 308Tell GN to use the testing server by enabling a build arg specific to the 309stm32f429i-disc1 target. 310 311.. code-block:: bash 312 313 $ gn args out 314 # Append this line to the file that opens in your editor to tell GN to run 315 # on-device unit tests. 316 pw_use_test_server = true 317 318Done! 319===== 320Whenever you make code changes and trigger a build, all the affected unit tests 321will be run across the attached boards! 322 323See the demo below for an example of what this all looks like put together: 324 325.. image:: https://storage.googleapis.com/pigweed-media/pw_watch_on_device_demo.gif 326 :width: 800 327 :alt: pw watch running on-device tests 328 329-------------------------- 330Building the documentation 331-------------------------- 332In addition to the markdown documentation, Pigweed has a collection of 333information-rich RST files that are used to generate HTML documentation. All 334the docs are hosted at https://pigweed.dev/, and are built as a part of the 335default build invocation. This makes it easier to make changes and see how they 336turn out. Once built, you can find the rendered HTML documentation at 337``out/docs/gen/docs/html``. 338 339You can explicitly build just the documentation with the command below. 340 341.. code-block:: bash 342 343 $ ninja -C out docs 344 345This concludes the introduction to developing for upstream Pigweed. 346 347--------------------------- 348Building tests individually 349--------------------------- 350Sometimes it's faster to incrementally build a single test target rather than 351waiting for the whole world to build and all tests to run. GN has a built-in 352tool, ``gn outputs``, that will translate a GN build step into a Ninja build 353step. In order to build and run the right test, it's important to explicitly 354specify which target to build the test under (e.g. host, SM32F529I-DISC1). 355This can be done by appending the GN path to the target toolchain in parenthesis 356after the desired GN build step label as seen in the example below. 357 358.. code-block:: none 359 360 $ gn outputs out "//pw_status:status_test.run(//targets/host/pigweed_internal:pw_strict_host_clang_debug)" 361 pw_strict_host_clang_debug/obj/pw_status/status_test.run.pw_pystamp 362 363 $ ninja -C out pw_strict_host_clang_debug/obj/pw_status/status_test.run.pw_pystamp 364 ninja: Entering directory `out' 365 [4/4] ACTION //pw_status:status_test.run(//targets/host/pigweed_internal:pw_strict_host_clang_debug) 366 367The ``.run`` following the test target name is a sub-target created as part of 368the ``pw_test`` GN template. If you remove ``.run``, the test will build but 369not attempt to run. 370 371In macOS and Linux, ``xargs`` can be used to turn this into a single command: 372 373.. code-block:: bash 374 375 $ gn outputs out "//pw_status:status_test.run(//targets/host/pigweed_internal:pw_strict_host_clang_debug)" | xargs ninja -C out 376 377---------- 378Next steps 379---------- 380 381Quickstarts 382=========== 383Visit :ref:`docs-get-started` to learn how to set up a new Bazel-based 384project, how to add Pigweed to an existing Bazel-based project, and more. 385 386Other modules 387============= 388If you'd like to see more of what Pigweed has to offer, dive into the 389:ref:`docs-module-guides`. 390 391The :ref:`docs-kudzu` repo demonstrates how to use Pigweed in your own project. 392Note that there are many ways to leverage Pigweed and Kudzu is just one 393approach. 394 395Editor setup 396============ 397Check out the :ref:`module-pw_ide` for setting up editor configurations or run 398the following for a quick setup: 399 400.. code-block:: bash 401 402 pw ide sync 403 404Hackaday Supercon talk about Pigweed 405==================================== 406We gave a talk at Hackaday's 2021 supercon, `Give Pigweed a Whirl 407<https://hackaday.com/2021/01/13/remoticon-video-pigweed-brings-embedded-unit-testing-library-integration-to-commandline/>`_ 408 409We've made improvements since we gave the talk; for example, we now have RTOS 410primitives. 411 412Get help 413======== 414Dropping into our `chat room <https://discord.gg/M9NSeTA>`_ is the most 415immediate way to get help from the Pigweed team. 416
