Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 1 | .. _module-pw_build-python: |
| 2 | |
| 3 | ------------------- |
| 4 | Python GN templates |
| 5 | ------------------- |
| 6 | The Python build is implemented with GN templates defined in |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 7 | ``pw_build/python.gni``. See the .gni file for complete usage documentation. |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 8 | |
| 9 | .. seealso:: :ref:`docs-python-build` |
| 10 | |
Anthony DiGirolamo | 211dce4 | 2021-08-12 16:48:43 -0700 | [diff] [blame] | 11 | .. _module-pw_build-pw_python_package: |
| 12 | |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 13 | pw_python_package |
| 14 | ================= |
| 15 | The main Python template is ``pw_python_package``. Each ``pw_python_package`` |
| 16 | target represents a Python package. As described in |
| 17 | :ref:`module-pw_build-python-target`, each ``pw_python_package`` expands to |
| 18 | several subtargets. In summary, these are: |
| 19 | |
| 20 | - ``<name>`` - Represents the files themselves |
| 21 | - ``<name>.lint`` - Runs static analysis |
| 22 | - ``<name>.tests`` - Runs all tests for this package |
| 23 | - ``<name>.install`` - Installs the package |
| 24 | - ``<name>.wheel`` - Builds a Python wheel |
| 25 | |
Wyatt Hepler | b85cda4 | 2021-04-07 13:13:15 -0700 | [diff] [blame] | 26 | GN permits using abbreviated labels when the target name matches the directory |
| 27 | name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, Python |
| 28 | package subtargets are aliased to the directory when the target name is the |
| 29 | same as the directory. For example, these two labels are equivalent: |
| 30 | |
| 31 | .. code-block:: |
| 32 | |
| 33 | //path/to/my_python_package:my_python_package.tests |
| 34 | //path/to/my_python_package:tests |
| 35 | |
Wyatt Hepler | f6f74f4 | 2021-04-13 19:26:20 -0700 | [diff] [blame] | 36 | The actions in a ``pw_python_package`` (e.g. installing packages and running |
| 37 | Pylint) are done within a single GN toolchain to avoid duplication in |
| 38 | multi-toolchain builds. This toolchain can be set with the |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 39 | ``pw_build_PYTHON_TOOLCHAIN`` GN arg, which defaults to |
| 40 | ``$dir_pw_build/python_toolchain:python``. |
Wyatt Hepler | f6f74f4 | 2021-04-13 19:26:20 -0700 | [diff] [blame] | 41 | |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 42 | Arguments |
| 43 | --------- |
| 44 | - ``setup`` - List of setup file paths (setup.py or pyproject.toml & setup.cfg), |
| 45 | which must all be in the same directory. |
Wyatt Hepler | b85cda4 | 2021-04-07 13:13:15 -0700 | [diff] [blame] | 46 | - ``generate_setup``: As an alternative to ``setup``, generate setup files with |
Anthony DiGirolamo | afb861c | 2021-08-12 16:48:04 -0700 | [diff] [blame] | 47 | the keywords in this scope. ``name`` is required. This follows the same |
| 48 | structure as a ``setup.cfg`` file's `declarative config |
| 49 | <https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html>`_ |
| 50 | For example: |
Wyatt Hepler | b85cda4 | 2021-04-07 13:13:15 -0700 | [diff] [blame] | 51 | |
| 52 | .. code-block:: |
| 53 | |
| 54 | generate_setup = { |
Anthony DiGirolamo | afb861c | 2021-08-12 16:48:04 -0700 | [diff] [blame] | 55 | metadata = { |
| 56 | name = "a_nifty_package" |
| 57 | version = "1.2a" |
| 58 | } |
| 59 | options = { |
| 60 | install_requires = [ "a_pip_package" ] |
| 61 | } |
Wyatt Hepler | b85cda4 | 2021-04-07 13:13:15 -0700 | [diff] [blame] | 62 | } |
| 63 | |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 64 | - ``sources`` - Python sources files in the package. |
| 65 | - ``tests`` - Test files for this Python package. |
| 66 | - ``python_deps`` - Dependencies on other pw_python_packages in the GN build. |
| 67 | - ``python_test_deps`` - Test-only pw_python_package dependencies. |
| 68 | - ``other_deps`` - Dependencies on GN targets that are not pw_python_packages. |
| 69 | - ``inputs`` - Other files to track, such as package_data. |
Wyatt Hepler | dcfcecf | 2021-03-01 08:36:19 -0800 | [diff] [blame] | 70 | - ``proto_library`` - A pw_proto_library target to embed in this Python package. |
Wyatt Hepler | b85cda4 | 2021-04-07 13:13:15 -0700 | [diff] [blame] | 71 | ``generate_setup`` is required in place of setup if proto_library is used. See |
Wyatt Hepler | dcfcecf | 2021-03-01 08:36:19 -0800 | [diff] [blame] | 72 | :ref:`module-pw_protobuf_compiler-add-to-python-package`. |
Wyatt Hepler | c2ce524 | 2021-03-17 08:42:14 -0700 | [diff] [blame] | 73 | - ``static_analysis`` List of static analysis tools to run; ``"*"`` (default) |
| 74 | runs all tools. The supported tools are ``"mypy"`` and ``"pylint"``. |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 75 | - ``pylintrc`` - Optional path to a pylintrc configuration file to use. If not |
| 76 | provided, Pylint's default rcfile search is used. Pylint is executed |
| 77 | from the package's setup directory, so pylintrc files in that directory |
| 78 | will take precedence over others. |
| 79 | - ``mypy_ini`` - Optional path to a mypy configuration file to use. If not |
| 80 | provided, mypy's default configuration file search is used. mypy is |
| 81 | executed from the package's setup directory, so mypy.ini files in that |
| 82 | directory will take precedence over others. |
| 83 | |
| 84 | Example |
| 85 | ------- |
| 86 | This is an example Python package declaration for a ``pw_my_module`` module. |
| 87 | |
| 88 | .. code-block:: |
| 89 | |
| 90 | import("//build_overrides/pigweed.gni") |
| 91 | |
| 92 | import("$dir_pw_build/python.gni") |
| 93 | |
| 94 | pw_python_package("py") { |
Anthony DiGirolamo | 7f8941f | 2021-08-24 10:02:04 -0700 | [diff] [blame] | 95 | setup = [ |
| 96 | "pyproject.toml", |
| 97 | "setup.cfg", |
| 98 | "setup.py", |
| 99 | ] |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 100 | sources = [ |
| 101 | "pw_my_module/__init__.py", |
| 102 | "pw_my_module/alfa.py", |
| 103 | "pw_my_module/bravo.py", |
| 104 | "pw_my_module/charlie.py", |
| 105 | ] |
| 106 | tests = [ |
| 107 | "alfa_test.py", |
| 108 | "charlie_test.py", |
| 109 | ] |
| 110 | python_deps = [ |
| 111 | "$dir_pw_status/py", |
| 112 | ":some_protos.python", |
| 113 | ] |
| 114 | python_test_deps = [ "$dir_pw_build/py" ] |
| 115 | pylintrc = "$dir_pigweed/.pylintrc" |
| 116 | } |
| 117 | |
| 118 | pw_python_script |
| 119 | ================ |
| 120 | A ``pw_python_script`` represents a set of standalone Python scripts and/or |
| 121 | tests. These files support all of the arguments of ``pw_python_package`` except |
| 122 | those ``setup``. These targets can be installed, but this only installs their |
| 123 | dependencies. |
| 124 | |
Wyatt Hepler | 473efe0 | 2021-05-04 14:15:16 -0700 | [diff] [blame] | 125 | ``pw_python_script`` allows creating a |
| 126 | :ref:`pw_python_action <module-pw_build-python-action>` associated with the |
| 127 | script. To create an action, pass an ``action`` scope to ``pw_python_script``. |
| 128 | If there is only a single source file, it serves as the action's ``script`` by |
| 129 | default. |
| 130 | |
| 131 | An action in ``pw_python_script`` can always be replaced with a standalone |
| 132 | ``pw_python_action``, but using the embedded action has some advantages: |
| 133 | |
| 134 | - The embedded action target bridges the gap between actions and Python targets. |
| 135 | A Python script can be expressed in a single, concise GN target, rather than |
| 136 | in two overlapping, dependent targets. |
| 137 | - The action automatically depends on the ``pw_python_script``. This ensures |
| 138 | that the script's dependencies are installed and the action automatically |
| 139 | reruns when the script's sources change, without needing to specify a |
| 140 | dependency, a step which is easy to forget. |
| 141 | - Using a ``pw_python_script`` with an embedded action is a simple way to check |
| 142 | an existing action's script with Pylint or Mypy or to add tests. |
| 143 | |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 144 | pw_python_group |
| 145 | =============== |
| 146 | Represents a group of ``pw_python_package`` and ``pw_python_script`` targets. |
| 147 | These targets do not add any files. Their subtargets simply forward to those of |
| 148 | their dependencies. |
| 149 | |
Wyatt Hepler | b3ea980 | 2021-02-23 09:46:09 -0800 | [diff] [blame] | 150 | pw_python_requirements |
| 151 | ====================== |
| 152 | Represents a set of local and PyPI requirements, with no associated source |
| 153 | files. These targets serve the role of a ``requirements.txt`` file. |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 154 | |
Michael Spang | 01faaf7 | 2021-06-09 22:38:40 -0400 | [diff] [blame] | 155 | When packages are installed by Pigweed, additional version constraints can be |
| 156 | provided using the ``pw_build_PIP_CONSTRAINTS`` GN arg. This option should |
| 157 | contain a list of paths to pass to the ``--constraint`` option of ``pip |
| 158 | install``. This can be used to synchronize dependency upgrades across a project |
| 159 | which facilitates reproducibility of builds. |
| 160 | |
| 161 | Note using multiple ``pw_python_requirements`` that install different versions |
| 162 | of the same package will currently cause unpredictable results, while using |
| 163 | constraints should have correct results (which may be an error indicating a |
| 164 | conflict). |
| 165 | |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 166 | .. _module-pw_build-python-dist: |
| 167 | |
| 168 | --------------------- |
Anthony DiGirolamo | 211dce4 | 2021-08-12 16:48:43 -0700 | [diff] [blame] | 169 | Python Distributables |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 170 | --------------------- |
| 171 | Pigweed also provides some templates to make it easier to bundle Python packages |
| 172 | for deployment. These templates are found in ``pw_build/python_dist.gni``. See |
Joe Ethier | b3ba70f | 2021-10-18 14:52:01 -0700 | [diff] [blame] | 173 | the .gni file for complete documentation on building distributables. |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 174 | |
| 175 | pw_python_wheels |
| 176 | ================ |
| 177 | Collects Python wheels for one or more ``pw_python_package`` targets, plus any |
| 178 | additional ``pw_python_package`` targets they depend on, directly or indirectly. |
| 179 | Note that this does not include Python dependencies that come from outside the |
| 180 | GN build, like packages from PyPI, for example. Those should still be declared |
| 181 | in the package's ``setup.py`` file as usual. |
| 182 | |
| 183 | Arguments |
| 184 | --------- |
| 185 | - ``packages`` - List of ``pw_python_package`` targets whose wheels should be |
| 186 | included; their dependencies will be pulled in as wheels also. |
Wyatt Hepler | 2b45465 | 2021-11-24 09:10:46 -0800 | [diff] [blame] | 187 | - ``directory`` - Output directory for the collected wheels. Defaults to |
| 188 | ``$target_out_dir/$target_name``. |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 189 | |
| 190 | Wheel collection under the hood |
| 191 | ------------------------------- |
| 192 | The ``.wheel`` subtarget of every ``pw_python_package`` generates a wheel |
| 193 | (``.whl``) for the Python package. The ``pw_python_wheels`` template figures |
| 194 | out which wheels to collect by traversing the ``pw_python_package_wheels`` |
| 195 | `GN metadata |
Rob Mohr | 463700d | 2021-05-22 10:31:03 -0700 | [diff] [blame] | 196 | <https://gn.googlesource.com/gn/+/HEAD/docs/reference.md#var_metadata>`_ key, |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 197 | which lists the output directory for each wheel. |
| 198 | |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 199 | pw_python_zip_with_setup |
| 200 | ======================== |
| 201 | Generates a ``.zip`` archive suitable for deployment outside of the project's |
| 202 | developer environment. The generated ``.zip`` contains Python wheels |
| 203 | (``.whl`` files) for one or more ``pw_python_package`` targets, plus wheels for |
| 204 | any additional ``pw_python_package`` targets in the GN build they depend on, |
| 205 | directly or indirectly. Dependencies from outside the GN build, such as packages |
Joe Ethier | b3ba70f | 2021-10-18 14:52:01 -0700 | [diff] [blame] | 206 | from PyPI, must be listed in packages' ``setup.py`` or ``setup.cfg`` files as |
| 207 | usual. |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 208 | |
| 209 | The ``.zip`` also includes simple setup scripts for Linux, |
| 210 | MacOS, and Windows. The setup scripts automatically create a Python virtual |
| 211 | environment and install the whole collection of wheels into it using ``pip``. |
| 212 | |
| 213 | Optionally, additional files and directories can be included in the archive. |
Joe Ethier | b3ba70f | 2021-10-18 14:52:01 -0700 | [diff] [blame] | 214 | One common example of an additional file to include is a README file with setup |
| 215 | and usage instructions for the distributable. A simple ready-to-use README file |
| 216 | is available at ``pw_build/py_dist/README.md``. |
Joe Ethier | fbe6615 | 2021-03-31 16:02:36 -0700 | [diff] [blame] | 217 | |
| 218 | Arguments |
| 219 | --------- |
| 220 | - ``packages`` - A list of `pw_python_package` targets whose wheels should be |
| 221 | included; their dependencies will be pulled in as wheels also. |
| 222 | - ``inputs`` - An optional list of extra files to include in the generated |
| 223 | ``.zip``, formatted the same way as the ``inputs`` argument to ``pw_zip`` |
| 224 | targets. |
| 225 | - ``dirs`` - An optional list of directories to include in the generated |
| 226 | ``.zip``, formatted the same was as the ``dirs`` argument to ``pw_zip`` |
| 227 | targets. |
| 228 | |
| 229 | Example |
| 230 | ------- |
| 231 | |
| 232 | .. code-block:: |
| 233 | |
| 234 | import("//build_overrides/pigweed.gni") |
| 235 | |
| 236 | import("$dir_pw_build/python_dist.gni") |
| 237 | |
| 238 | pw_python_zip_with_setup("my_tools") { |
| 239 | packages = [ ":some_python_package" ] |
| 240 | inputs = [ "$dir_pw_build/python_dist/README.md > /${target_name}/" ] |
| 241 | } |
Anthony DiGirolamo | 211dce4 | 2021-08-12 16:48:43 -0700 | [diff] [blame] | 242 | |
| 243 | pw_create_python_source_tree |
| 244 | ============================ |
| 245 | |
| 246 | Generates a directory of Python packages from source files suitable for |
| 247 | deployment outside of the project developer environment. The resulting directory |
| 248 | contains only files mentioned in each package's ``setup.cfg`` file. This is |
| 249 | useful for bundling multiple Python packages up into a single package for |
| 250 | distribution to other locations like `<http://pypi.org>`_. |
| 251 | |
| 252 | Arguments |
| 253 | --------- |
| 254 | |
| 255 | - ``packages`` - A list of :ref:`module-pw_build-pw_python_package` targets to be installed into |
| 256 | the build directory. Their dependencies will be pulled in as wheels also. |
| 257 | |
| 258 | - ``include_tests`` - If true, copy Python package tests to a ``tests`` subdir. |
| 259 | |
| 260 | - ``extra_files`` - A list of extra files that should be included in the output. |
| 261 | The format of each item in this list follows this convention: |
| 262 | |
| 263 | .. code-block:: text |
| 264 | |
| 265 | //some/nested/source_file > nested/destination_file |
| 266 | |
| 267 | - Source and destination file should be separated by ``>``. |
| 268 | |
| 269 | - The source file should be a GN target label (starting with ``//``). |
| 270 | |
| 271 | - The destination file will be relative to the generated output |
| 272 | directory. Parent directories are automatically created for each file. If a |
| 273 | file would be overwritten an error is raised. |
| 274 | |
Anthony DiGirolamo | b4517c3 | 2021-09-20 15:49:55 -0700 | [diff] [blame] | 275 | - ``generate_setup_cfg`` - If included, create a merged ``setup.cfg`` for all |
| 276 | python Packages using a ``common_config_file`` as a base. That file should |
| 277 | contain the required fields in the ``metadata`` and ``options`` sections as |
| 278 | shown in |
| 279 | `Configuring setup() using setup.cfg files <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html>`_. |
| 280 | ``append_git_sha_to_version`` and ``append_date_to_version`` will optionally |
| 281 | append the current git SHA or date to the package version string after a ``+`` |
| 282 | sign. |
| 283 | |
| 284 | .. code-block:: |
| 285 | |
| 286 | generate_setup_cfg = { |
| 287 | common_config_file = "pypi_common_setup.cfg" |
| 288 | append_git_sha_to_version = true |
| 289 | append_date_to_version = true |
| 290 | } |
Anthony DiGirolamo | 211dce4 | 2021-08-12 16:48:43 -0700 | [diff] [blame] | 291 | |
| 292 | Example |
| 293 | ------- |
| 294 | |
| 295 | :octicon:`file;1em` ./pw_env_setup/BUILD.gn |
| 296 | |
| 297 | .. code-block:: |
| 298 | |
| 299 | import("//build_overrides/pigweed.gni") |
| 300 | |
| 301 | import("$dir_pw_build/python_dist.gni") |
| 302 | |
| 303 | pw_create_python_source_tree("build_python_source_tree") { |
| 304 | packages = [ |
| 305 | ":some_python_package", |
| 306 | ":another_python_package", |
| 307 | ] |
| 308 | include_tests = true |
| 309 | extra_files = [ |
| 310 | "//README.md > ./README.md", |
| 311 | "//some_python_package/py/BUILD.bazel > some_python_package/BUILD.bazel", |
| 312 | "//another_python_package/py/BUILD.bazel > another_python_package/BUILD.bazel", |
| 313 | ] |
Anthony DiGirolamo | b4517c3 | 2021-09-20 15:49:55 -0700 | [diff] [blame] | 314 | generate_setup_cfg = { |
| 315 | common_config_file = "pypi_common_setup.cfg" |
| 316 | append_git_sha_to_version = true |
| 317 | append_date_to_version = true |
| 318 | } |
Anthony DiGirolamo | 211dce4 | 2021-08-12 16:48:43 -0700 | [diff] [blame] | 319 | } |
| 320 | |
| 321 | :octicon:`file-directory;1em` ./out/obj/pw_env_setup/build_python_source_tree/ |
| 322 | |
| 323 | .. code-block:: text |
| 324 | |
| 325 | $ tree ./out/obj/pw_env_setup/build_python_source_tree/ |
| 326 | ├── README.md |
Anthony DiGirolamo | b4517c3 | 2021-09-20 15:49:55 -0700 | [diff] [blame] | 327 | ├── setup.cfg |
Anthony DiGirolamo | 211dce4 | 2021-08-12 16:48:43 -0700 | [diff] [blame] | 328 | ├── some_python_package |
| 329 | │ ├── BUILD.bazel |
| 330 | │ ├── __init__.py |
| 331 | │ ├── py.typed |
| 332 | │ ├── some_source_file.py |
| 333 | │ └── tests |
| 334 | │ └── some_source_test.py |
| 335 | └── another_python_package |
| 336 | ├── BUILD.bazel |
| 337 | ├── __init__.py |
| 338 | ├── another_source_file.py |
| 339 | ├── py.typed |
| 340 | └── tests |
| 341 | └── another_source_test.py |