Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_unit_test: |
Alexei Frolov | 199045a | 2020-08-28 13:02:30 -0700 | [diff] [blame] | 2 | |
Alexei Frolov | ea39552 | 2020-03-13 13:35:07 -0700 | [diff] [blame] | 3 | ------------ |
| 4 | pw_unit_test |
| 5 | ------------ |
| 6 | ``pw_unit_test`` unit testing library with a `Google Test`_-compatible API, |
| 7 | built on top of embedded-friendly primitives. |
| 8 | |
Rob Mohr | 2f6662c | 2021-05-21 15:50:11 -0700 | [diff] [blame] | 9 | .. _Google Test: https://github.com/google/googletest/blob/HEAD/docs/primer.md |
Alexei Frolov | 8024679 | 2020-11-05 21:12:45 -0800 | [diff] [blame] | 10 | |
Alexei Frolov | ea39552 | 2020-03-13 13:35:07 -0700 | [diff] [blame] | 11 | ``pw_unit_test`` is a portable library which can run on almost any system from |
Shiva Rajagopal | 64d1096 | 2021-05-10 15:33:40 -0700 | [diff] [blame] | 12 | bare metal to a full-fledged desktop OS. It does this by offloading the |
Alexei Frolov | ea39552 | 2020-03-13 13:35:07 -0700 | [diff] [blame] | 13 | responsibility of test reporting and output to the underlying system, |
| 14 | communicating its results through a common interface. Unit tests can be written |
| 15 | once and run under many different environments, empowering developers to write |
| 16 | robust, high quality code. |
| 17 | |
| 18 | ``pw_unit_test`` is still under development and lacks many features expected in |
| 19 | a complete testing framework; nevertheless, it is already used heavily within |
| 20 | Pigweed. |
| 21 | |
| 22 | .. note:: |
| 23 | |
| 24 | This documentation is currently incomplete. |
| 25 | |
| 26 | Writing unit tests |
| 27 | ================== |
| 28 | |
| 29 | ``pw_unit_test``'s interface is largely compatible with `Google Test`_. Refer to |
| 30 | the Google Test documentation for examples of to define unit test cases. |
| 31 | |
| 32 | .. note:: |
| 33 | |
| 34 | A lot of Google Test's more advanced features are not yet implemented. To |
| 35 | request a feature addition, please |
Armando Montanez | 3d92e81 | 2020-03-19 12:13:36 -0700 | [diff] [blame] | 36 | `let us know <mailto:pigweed@googlegroups.com>`_. |
Alexei Frolov | ea39552 | 2020-03-13 13:35:07 -0700 | [diff] [blame] | 37 | |
| 38 | Using the test framework |
| 39 | ======================== |
| 40 | |
| 41 | The EventHandler interface |
| 42 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 43 | The ``EventHandler`` class in ``public/pw_unit_test/event_handler.h`` defines |
| 44 | the interface through which ``pw_unit_test`` communicates the results of its |
| 45 | test runs. A platform using ``pw_unit_test`` must register an event handler with |
| 46 | the unit testing framework to receive test output. |
| 47 | |
| 48 | As the framework runs tests, it calls the event handler's callback functions to |
| 49 | notify the system of various test events. The system can then choose to perform |
| 50 | any necessary handling or aggregation of these events, and report them back to |
| 51 | the developer. |
| 52 | |
| 53 | Predefined event handlers |
| 54 | ------------------------- |
| 55 | Pigweed provides some standard event handlers upstream to simplify the process |
| 56 | of getting started using ``pw_unit_test``. |
| 57 | |
| 58 | * ``SimplePrintingEventHandler``: An event handler that writes Google Test-style |
| 59 | output to a specified sink. |
| 60 | |
| 61 | .. code:: |
| 62 | |
| 63 | [==========] Running all tests. |
| 64 | [ RUN ] Status.Default |
| 65 | [ OK ] Status.Default |
| 66 | [ RUN ] Status.ConstructWithStatusCode |
| 67 | [ OK ] Status.ConstructWithStatusCode |
| 68 | [ RUN ] Status.AssignFromStatusCode |
| 69 | [ OK ] Status.AssignFromStatusCode |
| 70 | [ RUN ] Status.CompareToStatusCode |
| 71 | [ OK ] Status.CompareToStatusCode |
| 72 | [ RUN ] Status.Ok_OkIsTrue |
| 73 | [ OK ] Status.Ok_OkIsTrue |
| 74 | [ RUN ] Status.NotOk_OkIsFalse |
| 75 | [ OK ] Status.NotOk_OkIsFalse |
| 76 | [ RUN ] Status.KnownString |
| 77 | [ OK ] Status.KnownString |
| 78 | [ RUN ] Status.UnknownString |
| 79 | [ OK ] Status.UnknownString |
| 80 | [==========] Done running all tests. |
| 81 | [ PASSED ] 8 test(s). |
| 82 | |
| 83 | |
| 84 | * ``LoggingEventHandler``: An event handler which uses the ``pw_log`` module to |
| 85 | output test results, to integrate with the system's existing logging setup. |
| 86 | |
| 87 | .. _running-tests: |
| 88 | |
| 89 | Running tests |
| 90 | ^^^^^^^^^^^^^ |
| 91 | To run unit tests, link the tests into a single binary with the unit testing |
| 92 | framework, register an event handler, and call the ``RUN_ALL_TESTS`` macro. |
| 93 | |
| 94 | .. code:: cpp |
| 95 | |
| 96 | #include "pw_unit_test/framework.h" |
| 97 | #include "pw_unit_test/simple_printing_event_handler.h" |
| 98 | |
| 99 | void WriteString(const std::string_view& string, bool newline) { |
| 100 | printf("%s", string.data()); |
| 101 | if (newline) { |
| 102 | printf("\n"); |
| 103 | } |
| 104 | } |
| 105 | |
| 106 | int main() { |
| 107 | pw::unit_test::SimplePrintingEventHandler handler(WriteString); |
| 108 | pw::unit_test::RegisterEventHandler(&handler); |
| 109 | return RUN_ALL_TESTS(); |
| 110 | } |
| 111 | |
Alexei Frolov | 47a4304 | 2021-04-06 14:19:55 -0700 | [diff] [blame] | 112 | Test filtering |
| 113 | ^^^^^^^^^^^^^^ |
| 114 | If using C++17, filters can be set on the test framework to run only a subset of |
| 115 | the registered unit tests. This is useful when many tests are bundled into a |
| 116 | single application image. |
| 117 | |
| 118 | Currently, only a test suite filter is supported. This is set by calling |
| 119 | ``pw::unit_test::SetTestSuitesToRun`` with a list of suite names. |
| 120 | |
| 121 | .. note:: |
| 122 | Test filtering is only supported in C++17. |
| 123 | |
Alexei Frolov | ea39552 | 2020-03-13 13:35:07 -0700 | [diff] [blame] | 124 | Build system integration |
| 125 | ^^^^^^^^^^^^^^^^^^^^^^^^ |
| 126 | ``pw_unit_test`` integrates directly into Pigweed's GN build system. To define |
Armando Montanez | a761e32 | 2020-06-15 16:30:40 -0700 | [diff] [blame] | 127 | simple unit tests, set the ``pw_unit_test_MAIN`` build variable to a target |
Alexei Frolov | ea39552 | 2020-03-13 13:35:07 -0700 | [diff] [blame] | 128 | which configures the test framework as described in the :ref:`running-tests` |
| 129 | section, and use the ``pw_test`` template to register your test code. |
| 130 | |
| 131 | .. code:: |
| 132 | |
| 133 | import("$dir_pw_unit_test/test.gni") |
| 134 | |
| 135 | pw_test("foo_test") { |
| 136 | sources = [ "foo_test.cc" ] |
| 137 | } |
| 138 | |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 139 | The ``pw_unit_test`` module provides a few optional libraries to simplify setup: |
Armando Montanez | 0054a9b | 2020-03-13 13:06:24 -0700 | [diff] [blame] | 140 | |
Shiva Rajagopal | 64d1096 | 2021-05-10 15:33:40 -0700 | [diff] [blame] | 141 | - ``simple_printing_event_handler``: When running tests, output test results |
Armando Montanez | 0054a9b | 2020-03-13 13:06:24 -0700 | [diff] [blame] | 142 | as plain text over ``pw_sys_io``. |
| 143 | - ``simple_printing_main``: Implements a ``main()`` function that simply runs |
| 144 | tests using the ``simple_printing_event_handler``. |
| 145 | - ``logging_event_handler``: When running tests, log test results as |
| 146 | plain text using pw_log (ensure your target has set a ``pw_log`` backend). |
| 147 | - ``logging_main``: Implements a ``main()`` function that simply runs tests |
| 148 | using the ``logging_event_handler``. |
| 149 | |
| 150 | |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 151 | pw_test template |
| 152 | ---------------- |
Alexei Frolov | fbe68ff | 2020-11-16 13:44:52 -0800 | [diff] [blame] | 153 | ``pw_test`` defines a single unit test suite. It creates several sub-targets. |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 154 | |
Alexei Frolov | fbe68ff | 2020-11-16 13:44:52 -0800 | [diff] [blame] | 155 | * ``<target_name>``: The test suite within a single binary. The test code is |
| 156 | linked against the target set in the build arg ``pw_unit_test_MAIN``. |
| 157 | * ``<target_name>.run``: If ``pw_unit_test_AUTOMATIC_RUNNER`` is set, this |
| 158 | target runs the test as part of the build. |
| 159 | * ``<target_name>.lib``: The test sources without ``pw_unit_test_MAIN``. |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 160 | |
| 161 | **Arguments** |
| 162 | |
| 163 | * All GN executable arguments are accepted and forwarded to the underlying |
| 164 | ``pw_executable``. |
| 165 | * ``enable_if``: Boolean indicating whether the test should be built. If false, |
| 166 | replaces the test with an empty target. Default true. |
Michael Spang | 6aa8eb0 | 2020-06-11 20:19:05 -0400 | [diff] [blame] | 167 | * ``test_main``: Target label to add to the tests's dependencies to provide the |
| 168 | ``main()`` function. Defaults to ``pw_unit_test_MAIN``. Set to ``""`` if |
| 169 | ``main()`` is implemented in the test's ``sources``. |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 170 | |
| 171 | **Example** |
| 172 | |
| 173 | .. code:: |
| 174 | |
| 175 | import("$dir_pw_unit_test/test.gni") |
| 176 | |
| 177 | pw_test("large_test") { |
| 178 | sources = [ "large_test.cc" ] |
| 179 | enable_if = device_has_1m_flash |
| 180 | } |
| 181 | |
| 182 | |
| 183 | pw_test_group template |
| 184 | ---------------------- |
Alexei Frolov | fbe68ff | 2020-11-16 13:44:52 -0800 | [diff] [blame] | 185 | ``pw_test_group`` defines a collection of tests or other test groups. It creates |
| 186 | several sub-targets: |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 187 | |
Alexei Frolov | fbe68ff | 2020-11-16 13:44:52 -0800 | [diff] [blame] | 188 | * ``<target_name>``: The test group itself. |
| 189 | * ``<target_name>.run``: If ``pw_unit_test_AUTOMATIC_RUNNER`` is set, this |
| 190 | target runs all of the tests in the group and all of its group dependencies |
| 191 | individually. |
| 192 | * ``<target_name>.lib``: The sources of all of the tests in this group and its |
| 193 | dependencies. |
| 194 | * ``<target_name>.bundle``: All of the tests in the group and its dependencies |
| 195 | bundled into a single binary. |
| 196 | * ``<target_name>.bundle.run``: Automatic runner for the test bundle. |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 197 | |
| 198 | **Arguments** |
| 199 | |
| 200 | * ``tests``: List of the ``pw_test`` targets in the group. |
| 201 | * ``group_deps``: List of other ``pw_test_group`` targets on which this one |
| 202 | depends. |
| 203 | * ``enable_if``: Boolean indicating whether the group target should be created. |
| 204 | If false, an empty GN group is created instead. Default true. |
| 205 | |
| 206 | **Example** |
| 207 | |
| 208 | .. code:: |
| 209 | |
| 210 | import("$dir_pw_unit_test/test.gni") |
| 211 | |
| 212 | pw_test_group("tests") { |
| 213 | tests = [ |
| 214 | ":bar_test", |
| 215 | ":foo_test", |
| 216 | ] |
| 217 | } |
| 218 | |
| 219 | pw_test("foo_test") { |
| 220 | # ... |
| 221 | } |
| 222 | |
| 223 | pw_test("bar_test") { |
| 224 | # ... |
| 225 | } |
| 226 | |
Armando Montanez | 72b9385 | 2020-11-10 15:33:22 -0800 | [diff] [blame] | 227 | pw_facade_test template |
| 228 | ----------------------- |
| 229 | Pigweed facade test templates allow individual unit tests to build under the |
| 230 | current device target configuration while overriding specific build arguments. |
| 231 | This allows these tests to replace a facade's backend for the purpose of testing |
| 232 | the facade layer. |
| 233 | |
| 234 | .. warning:: |
| 235 | Facade tests are costly because each facade test will trigger a re-build of |
| 236 | every dependency of the test. While this sounds excessive, it's the only |
| 237 | technically correct way to handle this type of test. |
| 238 | |
| 239 | .. warning:: |
| 240 | Some facade test configurations may not be compatible with your target. Be |
| 241 | careful when running a facade test on a system that heavily depends on the |
| 242 | facade being tested. |
| 243 | |
Alexei Frolov | 8024679 | 2020-11-05 21:12:45 -0800 | [diff] [blame] | 244 | RPC service |
| 245 | =========== |
| 246 | ``pw_unit_test`` provides an RPC service which runs unit tests on demand and |
| 247 | streams the results back to the client. The service is defined in |
| 248 | ``pw_unit_test_proto/unit_test.proto``, and implemented by the GN target |
| 249 | ``$dir_pw_unit_test:rpc_service``. |
Alexei Frolov | bcac2c9 | 2020-05-19 10:24:20 -0700 | [diff] [blame] | 250 | |
Alexei Frolov | 8024679 | 2020-11-05 21:12:45 -0800 | [diff] [blame] | 251 | To set up RPC-based unit tests in your application, instantiate a |
| 252 | ``pw::unit_test::UnitTestService`` and register it with your RPC server. |
| 253 | |
| 254 | .. code:: c++ |
| 255 | |
| 256 | #include "pw_rpc/server.h" |
| 257 | #include "pw_unit_test/unit_test_service.h" |
| 258 | |
| 259 | // Server setup; refer to pw_rpc docs for more information. |
| 260 | pw::rpc::Channel channels[] = { |
| 261 | pw::rpc::Channel::Create<1>(&my_output), |
| 262 | }; |
| 263 | pw::rpc::Server server(channels); |
| 264 | |
| 265 | pw::unit_test::UnitTestService unit_test_service; |
| 266 | |
| 267 | void RegisterServices() { |
| 268 | server.RegisterService(unit_test_services); |
| 269 | } |
Alexei Frolov | 8cc1102 | 2020-12-07 15:03:44 -0800 | [diff] [blame] | 270 | |
| 271 | All tests flashed to an attached device can be run via python by calling |
| 272 | ``pw_unit_test.rpc.run_tests()`` with a RPC client services object that has |
| 273 | the unit testing RPC service enabled. By default, the results will output via |
| 274 | logging. |
| 275 | |
| 276 | .. code:: python |
| 277 | |
Alexei Frolov | d3e5cb7 | 2021-01-08 13:08:45 -0800 | [diff] [blame] | 278 | from pw_hdlc.rpc import HdlcRpcClient |
Alexei Frolov | 8cc1102 | 2020-12-07 15:03:44 -0800 | [diff] [blame] | 279 | from pw_unit_test.rpc import run_tests |
| 280 | |
| 281 | PROTO = Path(os.environ['PW_ROOT'], |
| 282 | 'pw_unit_test/pw_unit_test_proto/unit_test.proto') |
| 283 | |
| 284 | client = HdlcRpcClient(serial.Serial(device, baud), PROTO) |
| 285 | run_tests(client.rpcs()) |
Alexei Frolov | 47a4304 | 2021-04-06 14:19:55 -0700 | [diff] [blame] | 286 | |
| 287 | pw_unit_test.rpc |
| 288 | ^^^^^^^^^^^^^^^^ |
| 289 | .. automodule:: pw_unit_test.rpc |
| 290 | :members: EventHandler, run_tests |
Max Koopman | f478928 | 2021-05-14 11:01:55 -0700 | [diff] [blame] | 291 | |
| 292 | Module Configuration Options |
| 293 | ============================ |
| 294 | The following configurations can be adjusted via compile-time configuration of |
| 295 | this module. |
| 296 | |
| 297 | .. c:macro:: PW_UNIT_TEST_CONFIG_MEMORY_POOL_SIZE |
| 298 | |
| 299 | The size of the memory pool to use for test fixture instances. By default this |
| 300 | is set to 16K. |