Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_protobuf_compiler: |
Alexei Frolov | 4a257c1 | 2020-03-02 14:09:42 -0800 | [diff] [blame] | 2 | |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 3 | -------------------- |
| 4 | pw_protobuf_compiler |
| 5 | -------------------- |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 6 | The Protobuf compiler module provides build system integration and wrapper |
| 7 | scripts for generating source code for Protobuf definitions. |
| 8 | |
Alexei Frolov | f39cd8b | 2020-04-13 17:59:20 -0700 | [diff] [blame] | 9 | Generator support |
| 10 | ================= |
Alexei Frolov | f39cd8b | 2020-04-13 17:59:20 -0700 | [diff] [blame] | 11 | Protobuf code generation is currently supported for the following generators: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 12 | |
Alexei Frolov | 79b7cb0 | 2020-07-06 13:51:43 -0700 | [diff] [blame] | 13 | +-------------+----------------+-----------------------------------------------+ |
| 14 | | Generator | Code | Notes | |
| 15 | +-------------+----------------+-----------------------------------------------+ |
| 16 | | pw_protobuf | ``pwpb`` | Compiles using ``pw_protobuf``. | |
| 17 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | 79b7cb0 | 2020-07-06 13:51:43 -0700 | [diff] [blame] | 18 | | Nanopb | ``nanopb`` | Compiles using Nanopb. The build argument | |
| 19 | | | | ``dir_pw_third_party_nanopb`` must be set to | |
| 20 | | | | point to a local nanopb installation. | |
| 21 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | c912ea7 | 2020-10-26 08:43:27 -0700 | [diff] [blame] | 22 | | Nanopb RPC | ``nanopb_rpc`` | Compiles pw_rpc service and client code for | |
| 23 | | | | nanopb. Requires a nanopb installation. | |
| 24 | +-------------+----------------+-----------------------------------------------+ |
| 25 | | Raw RPC | ``raw_rpc`` | Compiles raw binary pw_rpc service code. | |
Alexei Frolov | 79b7cb0 | 2020-07-06 13:51:43 -0700 | [diff] [blame] | 26 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | a4c0aee | 2020-12-01 13:48:48 -0800 | [diff] [blame] | 27 | | Go | ``go`` | Compiles using the standard Go protobuf | |
| 28 | | | | plugin with gRPC service support. | |
| 29 | +-------------+----------------+-----------------------------------------------+ |
| 30 | | Python | ``python`` | Compiles using the standard Python protobuf | |
| 31 | | | | plugin, creating a ``pw_python_package``. | |
| 32 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 33 | |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 34 | GN template |
| 35 | =========== |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 36 | The ``pw_proto_library`` GN template is provided by the module. |
| 37 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 38 | It defines a collection of protobuf files that should be compiled together. The |
| 39 | template creates a sub-target for each supported generator, named |
| 40 | ``<target_name>.<generator>``. These sub-targets generate their respective |
| 41 | protobuf code, and expose it to the build system appropriately (e.g. a |
| 42 | ``pw_source_set`` for C/C++). |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 43 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 44 | For example, given the following target: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 45 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 46 | .. code-block:: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 47 | |
| 48 | pw_proto_library("test_protos") { |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 49 | sources = [ "my_test_protos/test.proto" ] |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 50 | } |
| 51 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 52 | ``test_protos.pwpb`` compiles code for pw_protobuf, and ``test_protos.nanopb`` |
| 53 | compiles using Nanopb (if it's installed). |
| 54 | |
| 55 | Protobuf code is only generated when a generator sub-target is listed as a |
| 56 | dependency of another GN target. |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 57 | |
Wyatt Hepler | 4cdc3e7 | 2021-03-24 08:41:08 -0700 | [diff] [blame] | 58 | GN permits using abbreviated labels when the target name matches the directory |
| 59 | name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, the |
| 60 | sub-targets for each generator are aliased to the directory when the target name |
| 61 | is the same. For example, these two labels are equivalent: |
| 62 | |
| 63 | .. code-block:: |
| 64 | |
| 65 | //path/to/my_protos:my_protos.pwpb |
| 66 | //path/to/my_protos:pwpb |
| 67 | |
Wyatt Hepler | 2c6c0ba | 2021-04-07 09:50:23 -0700 | [diff] [blame^] | 68 | ``pw_python_package`` subtargets are also available on the ``python`` subtarget: |
| 69 | |
| 70 | .. code-block:: |
| 71 | |
| 72 | //path/to/my_protos:my_protos.python.lint |
| 73 | //path/to/my_protos:python.lint |
| 74 | |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 75 | **Arguments** |
| 76 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 77 | * ``sources``: List of input .proto files. |
| 78 | * ``deps``: List of other pw_proto_library dependencies. |
| 79 | * ``inputs``: Other files on which the protos depend (e.g. nanopb ``.options`` |
| 80 | files). |
| 81 | * ``prefix``: A prefix to add to the source protos prior to compilation. For |
| 82 | example, a source called ``"foo.proto"`` with ``prefix = "nested"`` will be |
| 83 | compiled with protoc as ``"nested/foo.proto"``. |
| 84 | * ``strip_prefix``: Remove this prefix from the source protos. All source and |
| 85 | input files must be nested under this path. |
Wyatt Hepler | 4cdc3e7 | 2021-03-24 08:41:08 -0700 | [diff] [blame] | 86 | * ``python_package``: Label of Python package to which to add the proto modules. |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 87 | |
| 88 | **Example** |
| 89 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 90 | .. code-block:: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 91 | |
| 92 | import("$dir_pw_protobuf_compiler/proto.gni") |
| 93 | |
| 94 | pw_proto_library("my_protos") { |
| 95 | sources = [ |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 96 | "my_protos/foo.proto", |
| 97 | "my_protos/bar.proto", |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 98 | ] |
| 99 | } |
| 100 | |
| 101 | pw_proto_library("my_other_protos") { |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 102 | sources = [ "some/other/path/baz.proto" ] # imports foo.proto |
| 103 | |
| 104 | # This removes the "some/other/path" prefix from the proto files. |
| 105 | strip_prefix = "some/other/path" |
| 106 | |
| 107 | # This adds the "my_other_protos/" prefix to the proto files. |
| 108 | prefix = "my_other_protos" |
Alexei Frolov | 8e30d46 | 2020-10-22 13:54:36 -0700 | [diff] [blame] | 109 | |
| 110 | # Proto libraries depend on other proto libraries directly. |
| 111 | deps = [ ":my_protos" ] |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 112 | } |
| 113 | |
| 114 | source_set("my_cc_code") { |
| 115 | sources = [ |
| 116 | "foo.cc", |
| 117 | "bar.cc", |
| 118 | "baz.cc", |
| 119 | ] |
Alexei Frolov | 8e30d46 | 2020-10-22 13:54:36 -0700 | [diff] [blame] | 120 | |
| 121 | # When depending on protos in a source_set, specify the generator suffix. |
| 122 | deps = [ ":my_other_protos.pwpb" ] |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 123 | } |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 124 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 125 | From C++, ``baz.proto`` included as follows: |
| 126 | |
| 127 | .. code-block:: cpp |
| 128 | |
| 129 | #include "my_other_protos/baz.pwpb.h" |
| 130 | |
| 131 | From Python, ``baz.proto`` is imported as follows: |
| 132 | |
| 133 | .. code-block:: python |
| 134 | |
| 135 | from my_other_protos import baz_pb2 |
| 136 | |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 137 | Proto file structure |
| 138 | -------------------- |
| 139 | Protobuf source files must be nested under another directory when they are |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 140 | compiled. This ensures that they can be packaged properly in Python. The first |
| 141 | directory is used as the Python package name, so must be unique across the |
| 142 | build. The ``prefix`` option may be used to set this directory. |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 143 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 144 | Using ``prefix`` and ``strip_prefix`` together allows remapping proto files to |
| 145 | a completely different path. This can be useful when working with protos defined |
| 146 | in external libraries. For example, consider this proto library: |
| 147 | |
| 148 | .. code-block:: |
| 149 | |
| 150 | pw_proto_library("external_protos") { |
| 151 | sources = [ |
| 152 | "//other/external/some_library/src/protos/alpha.proto", |
| 153 | "//other/external/some_library/src/protos/beta.proto, |
| 154 | "//other/external/some_library/src/protos/internal/gamma.proto", |
| 155 | ] |
| 156 | strip_prefix = "//other/external/some_library/src/protos" |
| 157 | prefix = "some_library" |
| 158 | } |
| 159 | |
| 160 | These protos will be compiled by protoc as if they were in this file structure: |
| 161 | |
| 162 | .. code-block:: |
| 163 | |
| 164 | some_library/ |
| 165 | ├── alpha.proto |
| 166 | ├── beta.proto |
| 167 | └── internal |
| 168 | └── gamma.proto |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 169 | |
Wyatt Hepler | dcfcecf | 2021-03-01 08:36:19 -0800 | [diff] [blame] | 170 | .. _module-pw_protobuf_compiler-add-to-python-package: |
| 171 | |
| 172 | Adding Python proto modules to an existing package |
| 173 | -------------------------------------------------- |
| 174 | By default, generated Python proto modules are organized into their own Python |
| 175 | package. These proto modules can instead be added to an existing Python package |
Wyatt Hepler | b847615 | 2021-04-06 15:28:32 -0700 | [diff] [blame] | 176 | declared with ``pw_python_package``. This is done by setting the |
Wyatt Hepler | dcfcecf | 2021-03-01 08:36:19 -0800 | [diff] [blame] | 177 | ``python_package`` argument on the ``pw_proto_library`` and the |
| 178 | ``proto_library`` argument on the ``pw_python_package``. |
| 179 | |
| 180 | For example, the protos declared in ``my_protos`` will be nested in the Python |
| 181 | package declared by ``my_package``. |
| 182 | |
| 183 | .. code-block:: |
| 184 | |
| 185 | pw_proto_library("my_protos") { |
| 186 | sources = [ "hello.proto ] |
| 187 | prefix = "foo" |
| 188 | python_package = ":my_package" |
| 189 | } |
| 190 | |
| 191 | pw_python_pacakge("my_package") { |
| 192 | generate_setup = { |
| 193 | name = "foo" |
| 194 | version = "1.0" |
| 195 | } |
| 196 | sources = [ "foo/cool_module.py" ] |
| 197 | proto_library = ":my_protos" |
| 198 | } |
| 199 | |
| 200 | The ``hello_pb2.py`` proto module can be used alongside other files in the |
| 201 | ``foo`` package. |
| 202 | |
| 203 | .. code-block:: python |
| 204 | |
| 205 | from foo import cool_module, hello_pb2 |
| 206 | |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 207 | Working with externally defined protos |
| 208 | -------------------------------------- |
| 209 | ``pw_proto_library`` targets may be used to build ``.proto`` sources from |
| 210 | existing projects. In these cases, it may be necessary to supply the |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 211 | ``strip_prefix`` argument, which specifies the protobuf include path to use for |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 212 | ``protoc``. If only a single external protobuf is being compiled, the |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame] | 213 | ``python_module_as_package`` option can be used to override the requirement that |
| 214 | the protobuf be nested under a directory. This option generates a Python package |
| 215 | with the same name as the proto file, so that the generated proto can be |
| 216 | imported as if it were a standalone Python module. |
| 217 | |
| 218 | For example, the ``pw_proto_library`` target for Nanopb sets |
| 219 | ``python_module_as_package`` to ``nanopb_pb2``. |
| 220 | |
| 221 | .. code-block:: |
| 222 | |
| 223 | pw_proto_library("proto") { |
| 224 | strip_prefix = "$dir_pw_third_party_nanopb/generator/proto" |
| 225 | sources = [ "$dir_pw_third_party_nanopb/generator/proto/nanopb.proto" ] |
| 226 | python_module_as_package = "nanopb_pb2" |
| 227 | } |
| 228 | |
| 229 | In Python, this makes ``nanopb.proto`` available as ``import nanopb_pb2`` via |
| 230 | the ``nanopb_pb2`` Python package. In C++, ``nanopb.proto`` is accessed as |
| 231 | ``#include "nanopb.pwpb.h"``. |
| 232 | |
| 233 | The ``python_module_as_package`` feature should only be used when absolutely |
| 234 | necessary --- for example, to support proto files that include |
| 235 | ``import "nanopb.proto"``. |
| 236 | |
| 237 | CMake |
| 238 | ===== |
| 239 | CMake provides a ``pw_proto_library`` function with similar features as the |
| 240 | GN template. The CMake build only supports building firmware code, so |
| 241 | ``pw_proto_library`` does not generate a Python package. |
| 242 | |
| 243 | **Arguments** |
| 244 | |
| 245 | * ``NAME``: the base name of the libraries to create |
| 246 | * ``SOURCES``: .proto source files |
| 247 | * ``DEPS``: dependencies on other ``pw_proto_library`` targets |
| 248 | * ``PREFIX``: prefix add to the proto files |
| 249 | * ``STRIP_PREFIX``: prefix to remove from the proto files |
| 250 | * ``INPUTS``: files to include along with the .proto files (such as Nanopb |
| 251 | .options files) |
| 252 | |
| 253 | **Example** |
| 254 | |
| 255 | .. code-block:: cmake |
| 256 | |
| 257 | include($ENV{PW_ROOT}/pw_build/pigweed.cmake) |
| 258 | include($ENV{PW_ROOT}/pw_protobuf_compiler/proto.cmake) |
| 259 | |
| 260 | pw_proto_library(my_module.my_protos |
| 261 | SOURCES |
| 262 | my_protos/foo.proto |
| 263 | my_protos/bar.proto |
| 264 | ) |
| 265 | |
| 266 | pw_proto_library(my_module.my_protos |
| 267 | SOURCES |
| 268 | my_protos/foo.proto |
| 269 | my_protos/bar.proto |
| 270 | ) |
| 271 | |
| 272 | pw_proto_library(my_module.my_other_protos |
| 273 | SOURCES |
| 274 | some/other/path/baz.proto # imports foo.proto |
| 275 | |
| 276 | # This removes the "some/other/path" prefix from the proto files. |
| 277 | STRIP_PREFIX |
| 278 | some/other/path |
| 279 | |
| 280 | # This adds the "my_other_protos/" prefix to the proto files. |
| 281 | PREFIX |
| 282 | my_other_protos |
| 283 | |
| 284 | # Proto libraries depend on other proto libraries directly. |
| 285 | DEPS |
| 286 | my_module.my_protos |
| 287 | ) |
| 288 | |
| 289 | add_library(my_module.my_cc_code |
| 290 | foo.cc |
| 291 | bar.cc |
| 292 | baz.cc |
| 293 | ) |
| 294 | |
| 295 | # When depending on protos in a source_set, specify the generator suffix. |
| 296 | target_link_libraries(my_module.my_cc_code PUBLIC |
| 297 | my_module.my_other_protos.pwpb |
| 298 | ) |
| 299 | |
| 300 | These proto files are accessed in C++ the same as in the GN build: |
| 301 | |
| 302 | .. code-block:: cpp |
| 303 | |
Wyatt Hepler | 4cdc3e7 | 2021-03-24 08:41:08 -0700 | [diff] [blame] | 304 | #include "my_other_protos/baz.pwpb.h" |