1.. _module-pw_compilation_testing: 2 3====================== 4pw_compilation_testing 5====================== 6The pw_compilation_testing module provides for negative compilation (NC) 7testing. Negative compilation tests ensure that code that should not compile 8does not compile. Negative compilation testing is helpful in a variety of 9scenarios, for example: 10 11- Testing for compiler errors, such as ``[[nodiscard]]`` checks. 12- Testing that a template cannot be instantiated with certain types. 13- Testing that a ``static_assert`` statement is triggered as expected. 14- For a ``constexpr`` function, testing that a ``PW_ASSERT`` is triggered as 15 expected. 16 17Negative compilation tests are only supported in GN currently. Negative 18compilation tests are not currently supported in GN on Windows due to 19`b/241565082 <https://issues.pigweed.dev/241565082>`_. 20 21.. warning:: 22 23 This module is in an early, experimental state. Do not use it unless you have 24 consulted with the Pigweed team. 25 26--------------------------------- 27Negative compilation test example 28--------------------------------- 29.. code-block:: cpp 30 31 #include "pw_unit_test/framework.h" 32 #include "pw_compilation_testing/negative_compilation.h" 33 34 template <int kValue> 35 struct MyStruct { 36 static_assert(kValue % 2 == 0, "wrong number!"); 37 38 constexpr int MultiplyOdd(int runtime_value) const { 39 PW_ASSERT(runtime_value % 2 == 0); 40 return kValue * runtime_value; 41 } 42 }; 43 44 [[maybe_unused]] MyStruct<16> this_one_works; 45 46 // NC tests cannot be compiled, so they are created in preprocessor #if or 47 // #elif blocks. These NC tests check that a static_assert statement fails if 48 // the code is compiled. 49 #if PW_NC_TEST(NegativeOddNumber) 50 PW_NC_EXPECT("wrong number!"); 51 [[maybe_unused]] MyStruct<-1> illegal; 52 #elif PW_NC_TEST(PositiveOddNumber) 53 PW_NC_EXPECT("wrong number!"); 54 [[maybe_unused]] MyStruct<5> this_is_illegal; 55 #endif // PW_NC_TEST 56 57 struct Foo { 58 // Negative compilation tests can go anywhere in a source file. 59 #if PW_NC_TEST(IllegalValueAsClassMember) 60 PW_NC_EXPECT("wrong number!"); 61 MyStruct<12> also_illegal; 62 #endif // PW_NC_TEST 63 }; 64 65 TEST(MyStruct, MultiplyOdd) { 66 MyStruct<5> five; 67 EXPECT_EQ(five.MultiplyOdd(3), 15); 68 69 // This NC test checks that a specific PW_ASSERT() fails when expected. 70 // This only works in an NC test if the PW_ASSERT() fails while the compiler 71 // is executing constexpr code. The test code is used in a constexpr 72 // statement to force compile-time evaluation. 73 #if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber) 74 [[maybe_unused]] constexpr auto fail = [] { 75 PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);"); 76 MyStruct<3> my_struct; 77 return my_struct.MultiplyOdd(4); // Even number, PW_ASSERT should fail. 78 }(); 79 #endif // PW_NC_TEST 80 } 81 82 // PW_NC_TESTs can be conditionally executed using preprocessor conditionals. 83 #if PW_CXX_STANDARD_IS_SUPPORTED(20) 84 #if PW_NC_TEST(RequiresSomeCpp20Feature) 85 [[maybe_unused]] constinit MyStruct<4> constinit_works; 86 #endif // PW_NC_TEST 87 #endif // PW_CXX_STANDARD_IS_SUPPORTED(20) 88 89------------------------------------ 90Creating a negative compilation test 91------------------------------------ 92- Declare a ``pw_cc_negative_compilation_test()`` GN target or set 93 ``negative_compilation_test = true`` in a ``pw_test()`` target. 94- Add the test to the build in a toolchain with negative compilation testing 95 enabled (``pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true``). 96- In the test source files, add 97 ``#include "pw_compilation_testing/negative_compilation.h"``. 98- Use the ``PW_NC_TEST(TestName)`` macro in a ``#if`` statement. 99- Immediately after the ``PW_NC_TEST(TestName)``, provide one or more 100 Python-style regular expressions with the ``PW_NC_EXPECT()`` macro, one per 101 line. 102- Execute the tests by running the build. 103 104To simplify parsing, all ``PW_NC_TEST()`` statements must fit on a single line 105and cannot have any other code before or after them. ``PW_NC_EXPECT()`` 106statements may span multiple lines, but must contain a single regular expression 107as a string literal. The string may be comprised of multiple implicitly 108concatenated string literals. The ``PW_NC_EXPECT()`` statement cannot contain 109anything else except for ``//``-style comments. 110 111Test assertions 112=============== 113Negative compilation tests must have at least one assertion about the 114compilation output. The assertion macros must be placed immediately after the 115line with the ``PW_NC_TEST()`` or the test will fail. 116 117.. c:macro:: PW_NC_EXPECT(regex_string_literal) 118 119 When negative compilation tests are run, checks the compilation output for the 120 provided regular expression. The argument to the ``PW_NC_EXPECT()`` statement 121 must be a string literal. The literal is interpreted character-for-character 122 as a Python raw string literal and compiled as a Python `re 123 <https://docs.python.org/3/library/re.html>`_ regular expression. 124 125 For example, ``PW_NC_EXPECT("something (went|has gone) wrong!")`` searches the 126 failed compilation output with the Python regular expression 127 ``re.compile("something (went|has gone) wrong!")``. 128 129.. c:macro:: PW_NC_EXPECT_GCC(regex_string_literal) 130 131 Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with GCC. 132 133.. c:macro:: PW_NC_EXPECT_CLANG(regex_string_literal) 134 135 Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with Clang. 136 137.. admonition:: Test expectation tips 138 :class: tip 139 140 Be as specific as possible, but avoid compiler-specific error text. Try 141 matching against the following: 142 143 - ``static_assert`` messages. 144 - Contents of specific failing lines of source code: 145 ``PW_NC_EXPECT("PW_ASSERT\(!empty\(\));")``. 146 - Comments on affected lines: ``PW_NC_EXPECT("// Cannot construct from 147 nullptr")``. 148 - Function names: ``PW_NC_EXPECT("SomeFunction\(\).*private")``. 149 150 Do not match against the following: 151 152 - Source file paths. 153 - Source line numbers. 154 - Compiler-specific wording of error messages, except when necessary. 155 156------ 157Design 158------ 159The basic flow for negative compilation testing is as follows. 160 161- The user defines negative compilation tests in preprocessor ``#if`` blocks 162 using the ``PW_NC_TEST()`` and :c:macro:`PW_NC_EXPECT` macros. 163- The build invokes the ``pw_compilation_testing.generator`` script. The 164 generator script: 165 166 - finds ``PW_NC_TEST()`` statements and extracts a list of test cases, 167 - finds all associated :c:macro:`PW_NC_EXPECT` statements, and 168 - generates build targets for each negative compilation tests, 169 passing the test information and expectations to the targets. 170 171- The build compiles the test source file with all tests disabled. 172- The build invokes the negative compilation test targets, which run the 173 ``pw_compilation_testing.runner`` script. The test runner script: 174 175 - invokes the compiler, setting a preprocessor macro that enables the ``#if`` 176 block for the test. 177 - captures the compilation output, and 178 - checks the compilation output for the :c:macro:`PW_NC_EXPECT` expressions. 179 180- If compilation failed, and the output matches the test case's 181 :c:macro:`PW_NC_EXPECT` expressions, the test passes. 182- If compilation succeeded or the :c:macro:`PW_NC_EXPECT` expressions did not 183 match the output, the test fails. 184 185Existing frameworks 186=================== 187Pigweed's negative compilation tests were inspired by Chromium's `no-compile 188tests <https://www.chromium.org/developers/testing/no-compile-tests/>`_ 189tests and a similar framework used internally at Google. Pigweed's negative 190compilation testing framework improves on these systems in a few respects: 191 192- Trivial integration with unit tests. Negative compilation tests can easily be 193 placed alongside other unit tests instead of in separate files. 194- Safer, more natural macro-based API for test declarations. Other systems use 195 ``#ifdef`` macro checks to define test cases, which fail silently when there 196 are typos. Pigweed's framework uses function-like macros, which provide a 197 clean and natural API, catch typos, and ensure the test is integrated with the 198 NC test framework. 199- More readable, flexible test assertions. Other frameworks place assertions in 200 comments after test names, while Pigweed's framework uses function-like 201 macros. Pigweed also supports compiler-specific assertions. 202- Assertions are required. This helps ensure that compilation fails for the 203 expected reason and not for an accidental typo or unrelated issue. 204