1# Unit testing coreboot 2 3## Preface 4First part of this document, Introduction, comprises disambiguation for what 5unit testing is and what is not. This definition will be a basis for the whole 6paper. 7 8Next, Rationale, explains why to use unit testing and how coreboot specifically 9may benefit from it. 10 11This is followed by evaluation of different available free C unit test 12frameworks. Firstly, collection of requirements is provided. Secondly, there is 13a description of a few selected candidates. Finally, requirements are applied to 14candidates to see if they might be a good fit. 15 16Fourth part is a summary of evaluation, with proposal of unit test framework 17for coreboot to be used. 18 19Finally, Implementation proposal paragraph touches how build system and coreboot 20codebase in general should be organized, in order to support unit testing. This 21comprises couple of design considerations which need to be addressed. 22 23## Introduction 24A unit test is supposed to test a single unit of code in isolation. In C 25language (in contrary to OOP) unit usually means a function. One may also 26consider unit under test to be a single compilation unit which exposes some 27API (set of functions). A function, talking to some external component can be 28tested if this component can be mocked out. 29 30In other words (looking from C compilation angle), there should be no extra 31dependencies (executables) required beside unit under test and test harness in 32order to compile unit test binary. Test harness, beside code examining a 33routines, may comprise test framework implementation. 34 35It is hard to apply this strict definition of unit test to firmware code in 36practice, mostly due to constraints on speed of execution and size of final 37executable. coreboot codebase often cannot be adjusted to be testable. Because 38of this, coreboot unit testing subsystem should allow to include some additional 39source object files beside unit under test. That being said, the default and 40goal wherever possible, should be to isolate unit under test from other parts. 41 42Unit testing is not an integration testing and it doesn't replace it. First of 43all, integration tests cover larger set of components and interactions between 44them. Positive integration test result gives more confidence than a positive 45unit test does. Furthermore, unit tests are running on the build machine, while 46integration tests usually are executed on the target (or simulator). 47 48## Rationale 49Considering above, what is the benefit of unit testing, especially keeping in 50mind that coreboot is low-level firmware? Unit tests should be quick, thus may 51be executed frequently during development process. It is much easier to build 52and run a unit test on a build machine, than any integration test. This in turn 53may be used by dev to gather extra confidence early during code development 54process. Actually developer may even write unit tests earlier than the code - 55see [TDD](https://en.wikipedia.org/wiki/Test-driven_development) concept. 56 57That being said, unit testing embedded C code is a difficult task, due to 58significant amount of dependencies on underlying hardware. Mocking can handle 59some hardware dependencies. However, complex mocks make the unit test 60susceptible to failing and can require significant development effort. 61 62Writing unit tests for a code (both new and currently existing) may be favorable 63for the code quality. It is not only about finding bugs, but in general - easily 64testable code is a good code. 65 66coreboot benefits the most from testing common libraries (lib/, commonlib/, 67payloads/libpayload) and coreboot infrastructure (console/, device/, security/). 68 69## Evaluation of unit testing frameworks 70 71### Requirements 72Requirements for unit testing frameworks: 73 74* Easy to use 75* Few dependencies 76 77 Standard C library is all we should need 78 79* Isolation between tests 80* Support for mocking 81* Support for some machine parsable output 82* Compiler similarity 83 84 Compiler for the host _must_ support the same language standards as the target 85 compiler. Ideally the same toolchain should be used for building firmware 86 executables and test binaries, however the host compiler will be used to build 87 unit tests, whereas the coreboot toolchain will be used for building the 88 firmware executables. For some targets, the host compiler and the target 89 compiler could be the same, but this is not a requirement. 90 91* Same language for tests and code 92 93 Unit tests will be written in C, because coreboot code is also written in C 94 95### Desirables 96 97* Easy to integrate with build system/build tools 98 99 Ideally JUnit-like XML output format for Jenkins 100 101* Popularity is a plus 102 103 We want a larger community for a couple of reasons. Firstly, easier access to 104 people with knowledge and tutorials. Secondly, bug fixes for the top of tree 105 are more frequent and known issues are usually shorter in the pending state. 106 Last but not least, larger reviewer pool means better and easier upstream 107 improvements that we would like to submit. 108 109* Extra features may be a plus 110* Compatible license 111 112 This should not be a blocker, since test binaries are not distributed. 113 However ideally compatible with GPL. 114 115* IDE integration 116 117### Candidates 118There is a lot of frameworks which allow unit testing C code 119([list](https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C) from 120Wikipedia). While not all of them were evaluated, because that would take an 121excessive amount of time, couple of them were selected based on the good 122opinions among C devs, popularity and fitting above criteria. 123 124* [SputUnit](https://www.use-strict.de/sput-unit-testing/) 125* [GoogleTest](https://github.com/google/googletest) 126* [Cmocka](https://cmocka.org/) 127* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling) 128 129We looked at several other test frameworks, but decided not to do a full evaluation 130for various reasons such as functionality, size of the developer community, or 131compatibility. 132 133### Evaluation 134* [SputUnit](https://www.use-strict.de/sput-unit-testing/) 135 * Pros 136 * No dependencies, one header file to include - that’s all 137 * Pure C 138 * Very easy to use 139 * BSD license 140 * Cons 141 * Main repo doesn’t have support for generating JUnit XML reports for 142 Jenkins to consume - this feature is available only on the fork from 143 SputUnit called “Sput_report”. It makes it niche in a niche, so there are 144 some reservations whether support for this will be satisfactory 145 * No support for mocks 146 * Not too popular 147 * No automatic test registration 148* [GoogleTest](https://github.com/google/googletest) 149 * Pros 150 * Automatic test registration 151 * Support for different output formats (including XML for Jenkins) 152 * Good support, widely used, the biggest and the most active community out 153 of all frameworks that were investigated 154 * Available as a package in the most common distributions 155 * Test fixtures easily available 156 * Well documented 157 * Easy to integrate with an IDE 158 * BSD license 159 * Cons 160 * Requires C++11 compiler 161 * To make most out of it (use GMock) C++ knowledge is required 162* [Cmocka](https://cmocka.org/) 163 * Pros 164 * Self-contained, autonomous framework 165 * Pure C 166 * API is well documented 167 * Multiple output formats (including XML for Jenkins) 168 * Available as a package in the most common distributions 169 * Used in some popular open source projects (libssh, OpenVPN, Samba) 170 * Test fixtures available 171 * Support for exception handling 172 * Cons 173 * No automatic test registration 174 * It will require some effort to make it work from within an IDE 175 * Apache 2.0 license (not compatible with GPLv2) 176* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling) 177 * Pros 178 * Pure C (Unity testing framework itself, not test runner) 179 * Support for different output formats (including XML for Jenkins) 180 * There are some (rather easy) hints how to use this from an IDE (e.g. Eclipse) 181 * MIT license 182 * Cons 183 * Test runner (Ceedling) is not written in C - uses Ruby 184 * Mocking/Exception handling functionalities are actually separate tools 185 * No automatic test registration 186 * Not too popular 187 188### Summary & framework proposal 189After research, we propose using the Cmocka unit test framework. Cmocka fulfills 190all stated evaluation criteria. It is rather easy to use, doesn’t have extra 191dependencies, written fully in C, allows for tests fixtures and some popular 192open source projects already are using it. Cmocka also includes support for 193mocks. 194 195Cmocka's limitations, such as the lack of automatic test registration, are 196considered minor issues that will require only minimal additional work from a 197developer. At the same time, it may be worth to propose improvement to Cmocka 198community or simply apply some extra wrapper with demanded functionality. 199 200## Implementation 201 202### Framework as a submodule or external package 203Unit test frameworks may be either compiled from source (from a git submodule 204under 3rdparty/) or pre-compiled as a package. The second option seems to be 205easier to maintain, while at the same time may bring some unwanted consequences 206(different version across distributions, frequent changes in API). It makes sense 207to initially experiment with packages and check how it works. If this will 208cause any issues, then it is always possible to switch to submodule approach. 209 210### Integration with build system 211To get the most out of unit testing framework, it should be integrated with 212Jenkins automation server. Verification of all unit tests for new changes may 213improve code reliability to some extent. 214 215### Build configuration (Kconfig) 216While building unit under test object file, it is necessary to apply some 217configuration (config) just like when building usual firmware. For simplicity, 218there will be one default tests .config `qemu_x86_i440fx` for all unit tests. At 219the same time, some tests may require running with different values of particular 220config. This should be handled by adding extra header, included after config.h. 221This header will comprise #undef of old CONFIG values and #define of the 222required value. When unit testing will be integrated with Jenkins, it may be 223preferred to use every available config for periodic builds. 224 225### Directory structure 226Tests should be kept separate from the code, while at the same time it must be 227easy to match code with test harness. 228 229We create new directory for test files ($(toplevel)/tests/) and mimic the 230structure of src/ directory. 231 232Test object files (test harness, unit under tests and any additional executables 233are stored under build/tests/<test_name> directory. 234 235Below example shows how directory structure is organized for the two test cases: 236tests/lib/string-test and tests/device/i2c-test: 237 238```bash 239├── src 240│ ├── lib 241│ │ ├── string.c <- unit under test 242│ │ 243│ ├── device 244│ ├── i2c.c 245│ 246├── tests 247│ ├── include 248│ │ ├── mocks <- mock headers, which replace original headers 249│ │ 250│ ├── Makefile.mk <- top Makefile for unit tests subsystem 251│ ├── lib 252│ │ ├── Makefile.mk 253│ │ ├── string-test.c <- test code for src/lib/string.c 254│ │ │ 255│ ├── device 256│ │ ├── Makefile.mk 257│ ├── i2c-test.c 258│ 259├── build 260│ ├── tests <-all test-related executables 261 ├── config.h <- default config used for tests builds 262 ├── lib 263 │ ├── string-test <- all string-test executables 264 │ │ ├── run <- final test binary 265 │ │ ├── tests <- all test harness executables 266 │ │ ├── lib 267 │ │ ├── string-test.o <-test harness executable 268 │ │ ├── src <- unit under test and other src executables 269 │ │ ├── lib 270 │ │ ├── string.o <- unit under test executable 271 ├── device 272 ├── i2c-test 273 ├── run 274 ├── tests 275 │ ├── device 276 │ ├── i2c-test.o 277 ├── src 278 ├── device 279 ├── i2c.o 280``` 281 282### Writing new tests 283Our tutorial series has [detailed guidelines](../tutorial/part3.md) for writing 284unit tests. 285