Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_protobuf_compiler: |
Alexei Frolov | 4a257c1 | 2020-03-02 14:09:42 -0800 | [diff] [blame] | 2 | |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 3 | -------------------- |
| 4 | pw_protobuf_compiler |
| 5 | -------------------- |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 6 | The Protobuf compiler module provides build system integration and wrapper |
| 7 | scripts for generating source code for Protobuf definitions. |
| 8 | |
Alexei Frolov | f39cd8b | 2020-04-13 17:59:20 -0700 | [diff] [blame] | 9 | Generator support |
| 10 | ================= |
Alexei Frolov | f39cd8b | 2020-04-13 17:59:20 -0700 | [diff] [blame] | 11 | Protobuf code generation is currently supported for the following generators: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 12 | |
Alexei Frolov | 79b7cb0 | 2020-07-06 13:51:43 -0700 | [diff] [blame] | 13 | +-------------+----------------+-----------------------------------------------+ |
| 14 | | Generator | Code | Notes | |
| 15 | +-------------+----------------+-----------------------------------------------+ |
| 16 | | pw_protobuf | ``pwpb`` | Compiles using ``pw_protobuf``. | |
| 17 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | 79b7cb0 | 2020-07-06 13:51:43 -0700 | [diff] [blame] | 18 | | Nanopb | ``nanopb`` | Compiles using Nanopb. The build argument | |
| 19 | | | | ``dir_pw_third_party_nanopb`` must be set to | |
| 20 | | | | point to a local nanopb installation. | |
| 21 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | c912ea7 | 2020-10-26 08:43:27 -0700 | [diff] [blame] | 22 | | Nanopb RPC | ``nanopb_rpc`` | Compiles pw_rpc service and client code for | |
| 23 | | | | nanopb. Requires a nanopb installation. | |
| 24 | +-------------+----------------+-----------------------------------------------+ |
| 25 | | Raw RPC | ``raw_rpc`` | Compiles raw binary pw_rpc service code. | |
Alexei Frolov | 79b7cb0 | 2020-07-06 13:51:43 -0700 | [diff] [blame] | 26 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | a4c0aee | 2020-12-01 13:48:48 -0800 | [diff] [blame] | 27 | | Go | ``go`` | Compiles using the standard Go protobuf | |
| 28 | | | | plugin with gRPC service support. | |
| 29 | +-------------+----------------+-----------------------------------------------+ |
| 30 | | Python | ``python`` | Compiles using the standard Python protobuf | |
| 31 | | | | plugin, creating a ``pw_python_package``. | |
| 32 | +-------------+----------------+-----------------------------------------------+ |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 33 | |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 34 | GN template |
| 35 | =========== |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 36 | The ``pw_proto_library`` GN template is provided by the module. |
| 37 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 38 | It defines a collection of protobuf files that should be compiled together. The |
| 39 | template creates a sub-target for each supported generator, named |
| 40 | ``<target_name>.<generator>``. These sub-targets generate their respective |
| 41 | protobuf code, and expose it to the build system appropriately (e.g. a |
| 42 | ``pw_source_set`` for C/C++). |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 43 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 44 | For example, given the following target: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 45 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 46 | .. code-block:: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 47 | |
| 48 | pw_proto_library("test_protos") { |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 49 | sources = [ "my_test_protos/test.proto" ] |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 50 | } |
| 51 | |
Alexei Frolov | b499d3f | 2020-10-28 13:00:08 -0700 | [diff] [blame] | 52 | ``test_protos.pwpb`` compiles code for pw_protobuf, and ``test_protos.nanopb`` |
| 53 | compiles using Nanopb (if it's installed). |
| 54 | |
| 55 | Protobuf code is only generated when a generator sub-target is listed as a |
| 56 | dependency of another GN target. |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 57 | |
| 58 | **Arguments** |
| 59 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 60 | * ``sources``: List of input .proto files. |
| 61 | * ``deps``: List of other pw_proto_library dependencies. |
| 62 | * ``inputs``: Other files on which the protos depend (e.g. nanopb ``.options`` |
| 63 | files). |
| 64 | * ``prefix``: A prefix to add to the source protos prior to compilation. For |
| 65 | example, a source called ``"foo.proto"`` with ``prefix = "nested"`` will be |
| 66 | compiled with protoc as ``"nested/foo.proto"``. |
| 67 | * ``strip_prefix``: Remove this prefix from the source protos. All source and |
| 68 | input files must be nested under this path. |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 69 | |
| 70 | **Example** |
| 71 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 72 | .. code-block:: |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 73 | |
| 74 | import("$dir_pw_protobuf_compiler/proto.gni") |
| 75 | |
| 76 | pw_proto_library("my_protos") { |
| 77 | sources = [ |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 78 | "my_protos/foo.proto", |
| 79 | "my_protos/bar.proto", |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 80 | ] |
| 81 | } |
| 82 | |
| 83 | pw_proto_library("my_other_protos") { |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 84 | sources = [ "some/other/path/baz.proto" ] # imports foo.proto |
| 85 | |
| 86 | # This removes the "some/other/path" prefix from the proto files. |
| 87 | strip_prefix = "some/other/path" |
| 88 | |
| 89 | # This adds the "my_other_protos/" prefix to the proto files. |
| 90 | prefix = "my_other_protos" |
Alexei Frolov | 8e30d46 | 2020-10-22 13:54:36 -0700 | [diff] [blame] | 91 | |
| 92 | # Proto libraries depend on other proto libraries directly. |
| 93 | deps = [ ":my_protos" ] |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 94 | } |
| 95 | |
| 96 | source_set("my_cc_code") { |
| 97 | sources = [ |
| 98 | "foo.cc", |
| 99 | "bar.cc", |
| 100 | "baz.cc", |
| 101 | ] |
Alexei Frolov | 8e30d46 | 2020-10-22 13:54:36 -0700 | [diff] [blame] | 102 | |
| 103 | # When depending on protos in a source_set, specify the generator suffix. |
| 104 | deps = [ ":my_other_protos.pwpb" ] |
Alexei Frolov | 942adf0 | 2019-12-11 17:07:28 -0800 | [diff] [blame] | 105 | } |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 106 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 107 | From C++, ``baz.proto`` included as follows: |
| 108 | |
| 109 | .. code-block:: cpp |
| 110 | |
| 111 | #include "my_other_protos/baz.pwpb.h" |
| 112 | |
| 113 | From Python, ``baz.proto`` is imported as follows: |
| 114 | |
| 115 | .. code-block:: python |
| 116 | |
| 117 | from my_other_protos import baz_pb2 |
| 118 | |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 119 | Proto file structure |
| 120 | -------------------- |
| 121 | Protobuf source files must be nested under another directory when they are |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 122 | compiled. This ensures that they can be packaged properly in Python. The first |
| 123 | directory is used as the Python package name, so must be unique across the |
| 124 | build. The ``prefix`` option may be used to set this directory. |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 125 | |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 126 | Using ``prefix`` and ``strip_prefix`` together allows remapping proto files to |
| 127 | a completely different path. This can be useful when working with protos defined |
| 128 | in external libraries. For example, consider this proto library: |
| 129 | |
| 130 | .. code-block:: |
| 131 | |
| 132 | pw_proto_library("external_protos") { |
| 133 | sources = [ |
| 134 | "//other/external/some_library/src/protos/alpha.proto", |
| 135 | "//other/external/some_library/src/protos/beta.proto, |
| 136 | "//other/external/some_library/src/protos/internal/gamma.proto", |
| 137 | ] |
| 138 | strip_prefix = "//other/external/some_library/src/protos" |
| 139 | prefix = "some_library" |
| 140 | } |
| 141 | |
| 142 | These protos will be compiled by protoc as if they were in this file structure: |
| 143 | |
| 144 | .. code-block:: |
| 145 | |
| 146 | some_library/ |
| 147 | ├── alpha.proto |
| 148 | ├── beta.proto |
| 149 | └── internal |
| 150 | └── gamma.proto |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 151 | |
| 152 | Working with externally defined protos |
| 153 | -------------------------------------- |
| 154 | ``pw_proto_library`` targets may be used to build ``.proto`` sources from |
| 155 | existing projects. In these cases, it may be necessary to supply the |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 156 | ``strip_prefix`` argument, which specifies the protobuf include path to use for |
Wyatt Hepler | d517afc | 2021-02-03 19:40:08 -0800 | [diff] [blame] | 157 | ``protoc``. If only a single external protobuf is being compiled, the |
Wyatt Hepler | 752d7d3 | 2021-03-02 09:02:23 -0800 | [diff] [blame^] | 158 | ``python_module_as_package`` option can be used to override the requirement that |
| 159 | the protobuf be nested under a directory. This option generates a Python package |
| 160 | with the same name as the proto file, so that the generated proto can be |
| 161 | imported as if it were a standalone Python module. |
| 162 | |
| 163 | For example, the ``pw_proto_library`` target for Nanopb sets |
| 164 | ``python_module_as_package`` to ``nanopb_pb2``. |
| 165 | |
| 166 | .. code-block:: |
| 167 | |
| 168 | pw_proto_library("proto") { |
| 169 | strip_prefix = "$dir_pw_third_party_nanopb/generator/proto" |
| 170 | sources = [ "$dir_pw_third_party_nanopb/generator/proto/nanopb.proto" ] |
| 171 | python_module_as_package = "nanopb_pb2" |
| 172 | } |
| 173 | |
| 174 | In Python, this makes ``nanopb.proto`` available as ``import nanopb_pb2`` via |
| 175 | the ``nanopb_pb2`` Python package. In C++, ``nanopb.proto`` is accessed as |
| 176 | ``#include "nanopb.pwpb.h"``. |
| 177 | |
| 178 | The ``python_module_as_package`` feature should only be used when absolutely |
| 179 | necessary --- for example, to support proto files that include |
| 180 | ``import "nanopb.proto"``. |
| 181 | |
| 182 | CMake |
| 183 | ===== |
| 184 | CMake provides a ``pw_proto_library`` function with similar features as the |
| 185 | GN template. The CMake build only supports building firmware code, so |
| 186 | ``pw_proto_library`` does not generate a Python package. |
| 187 | |
| 188 | **Arguments** |
| 189 | |
| 190 | * ``NAME``: the base name of the libraries to create |
| 191 | * ``SOURCES``: .proto source files |
| 192 | * ``DEPS``: dependencies on other ``pw_proto_library`` targets |
| 193 | * ``PREFIX``: prefix add to the proto files |
| 194 | * ``STRIP_PREFIX``: prefix to remove from the proto files |
| 195 | * ``INPUTS``: files to include along with the .proto files (such as Nanopb |
| 196 | .options files) |
| 197 | |
| 198 | **Example** |
| 199 | |
| 200 | .. code-block:: cmake |
| 201 | |
| 202 | include($ENV{PW_ROOT}/pw_build/pigweed.cmake) |
| 203 | include($ENV{PW_ROOT}/pw_protobuf_compiler/proto.cmake) |
| 204 | |
| 205 | pw_proto_library(my_module.my_protos |
| 206 | SOURCES |
| 207 | my_protos/foo.proto |
| 208 | my_protos/bar.proto |
| 209 | ) |
| 210 | |
| 211 | pw_proto_library(my_module.my_protos |
| 212 | SOURCES |
| 213 | my_protos/foo.proto |
| 214 | my_protos/bar.proto |
| 215 | ) |
| 216 | |
| 217 | pw_proto_library(my_module.my_other_protos |
| 218 | SOURCES |
| 219 | some/other/path/baz.proto # imports foo.proto |
| 220 | |
| 221 | # This removes the "some/other/path" prefix from the proto files. |
| 222 | STRIP_PREFIX |
| 223 | some/other/path |
| 224 | |
| 225 | # This adds the "my_other_protos/" prefix to the proto files. |
| 226 | PREFIX |
| 227 | my_other_protos |
| 228 | |
| 229 | # Proto libraries depend on other proto libraries directly. |
| 230 | DEPS |
| 231 | my_module.my_protos |
| 232 | ) |
| 233 | |
| 234 | add_library(my_module.my_cc_code |
| 235 | foo.cc |
| 236 | bar.cc |
| 237 | baz.cc |
| 238 | ) |
| 239 | |
| 240 | # When depending on protos in a source_set, specify the generator suffix. |
| 241 | target_link_libraries(my_module.my_cc_code PUBLIC |
| 242 | my_module.my_other_protos.pwpb |
| 243 | ) |
| 244 | |
| 245 | These proto files are accessed in C++ the same as in the GN build: |
| 246 | |
| 247 | .. code-block:: cpp |
| 248 | |
| 249 | #include "my_other_protos/baz.pwpb" |