Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_env_setup: |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 2 | |
| 3 | ------------ |
| 4 | pw_env_setup |
| 5 | ------------ |
| 6 | A classic problem in the embedded space is reducing the time from git clone |
| 7 | to having a binary executing on a device. The issue is that an entire suite |
| 8 | of tools is needed for non-trivial production embedded projects. For example: |
| 9 | |
| 10 | - A C++ compiler for your target device, and also for your host |
| 11 | - A build system or three; for example, GN, Ninja, CMake, Bazel |
| 12 | - A code formatting program like clang-format |
| 13 | - A debugger like OpenOCD to flash and debug your embedded device |
| 14 | - A known Python version with known modules installed for scripting |
| 15 | - A Go compiler for the Go-based command line tools |
| 16 | |
| 17 | ...and so on |
| 18 | |
| 19 | In the server space, container solutions like Docker or Podman solve this; |
| 20 | however, in our experience container solutions are a mixed bag for embedded |
| 21 | systems development where one frequently needs access to native system |
| 22 | resources like USB devices, or must operate on Windows. |
| 23 | |
Armando Montanez | 0054a9b | 2020-03-13 13:06:24 -0700 | [diff] [blame] | 24 | ``pw_env_setup`` is our compromise solution for this problem that works on Mac, |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 25 | Windows, and Linux. It leverages the Chrome packaging system `CIPD`_ to |
| 26 | bootstrap a Python installation, which in turn inflates a virtual |
| 27 | environment. The tooling is installed into your workspace, and makes no |
| 28 | changes to your system. This tooling is designed to be reused by any |
| 29 | project. |
| 30 | |
Rob Mohr | 55bb0ad | 2021-05-22 10:59:52 -0700 | [diff] [blame] | 31 | .. _CIPD: https://github.com/luci/luci-go/tree/HEAD/cipd |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 32 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 33 | Users interact with ``pw_env_setup`` with two commands: ``. bootstrap.sh`` and |
| 34 | ``. activate.sh``. The bootstrap command always pulls down the current versions |
| 35 | of CIPD packages and sets up the Python virtual environment. The activate |
| 36 | command reinitializes a previously configured environment, and if none is found, |
| 37 | runs bootstrap. |
| 38 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 39 | .. note:: |
| 40 | On Windows the scripts used to set up the environment are ``bootstrap.bat`` |
| 41 | and ``activate.bat``. For simplicity they will be referred to with the ``.sh`` |
| 42 | endings unless the distinction is relevant. |
| 43 | |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 44 | .. warning:: |
Armando Montanez | 0054a9b | 2020-03-13 13:06:24 -0700 | [diff] [blame] | 45 | At this time ``pw_env_setup`` works for us, but isn’t well tested. We don’t |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 46 | suggest relying on it just yet. However, we are interested in experience |
| 47 | reports; if you give it a try, please `send us a note`_ about your |
| 48 | experience. |
| 49 | |
| 50 | .. _send us a note: pigweed@googlegroups.com |
Rob Mohr | d596a53 | 2020-04-03 08:59:12 -0700 | [diff] [blame] | 51 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 52 | ================================== |
| 53 | Using pw_env_setup in your project |
| 54 | ================================== |
| 55 | |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 56 | Downstream Projects Using Pigweed's Packages |
| 57 | ******************************************** |
Rob Mohr | d596a53 | 2020-04-03 08:59:12 -0700 | [diff] [blame] | 58 | |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 59 | Projects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's |
| 60 | dependencies or their own dependencies. Projects that only want to use Pigweed's |
| 61 | dependencies without modifying them can just source Pigweed's ``bootstrap.sh`` |
| 62 | and ``activate.sh`` scripts. |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 63 | |
| 64 | An example of what your project's `bootstrap.sh` could look like is below. This |
| 65 | assumes `bootstrap.sh` is at the top level of your repository. |
| 66 | |
| 67 | .. code-block:: bash |
| 68 | |
| 69 | # Do not include a "#!" line, this must be sourced and not executed. |
| 70 | |
| 71 | # This assumes the user is sourcing this file from it's parent directory. See |
| 72 | # below for a more flexible way to handle this. |
| 73 | PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh" |
| 74 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 75 | export PW_PROJECT_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")" |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 76 | |
| 77 | # You may wish to check if the user is attempting to execute this script |
| 78 | # instead of sourcing it. See below for an example of how to handle that |
| 79 | # situation. |
| 80 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 81 | # Source Pigweed's bootstrap utility script. |
| 82 | # Using '.' instead of 'source' for POSIX compatibility. Since users don't use |
| 83 | # dash directly, using 'source' in most documentation so users don't get |
| 84 | # confused and try to `./bootstrap.sh`. |
| 85 | . "$PW_PROJECT_ROOT/third_party/pigweed/pw_env_setup/util.sh" |
| 86 | |
| 87 | pw_check_root "$PW_ROOT" |
| 88 | _PW_ACTUAL_ENVIRONMENT_ROOT="$(pw_get_env_root)" |
| 89 | export _PW_ACTUAL_ENVIRONMENT_ROOT |
| 90 | SETUP_SH="$_PW_ACTUAL_ENVIRONMENT_ROOT/activate.sh" |
| 91 | pw_bootstrap --args... # See below for details about args. |
| 92 | pw_finalize bootstrap "$SETUP_SH" |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 93 | |
Nathaniel Brough | e22ac20 | 2021-04-16 10:30:37 +0800 | [diff] [blame] | 94 | |
| 95 | Bazel Usage |
| 96 | ----------- |
| 97 | It is possible to pull in a CIPD dependency into Bazel using WORKSPACE rules |
| 98 | rather than using `bootstrap.sh`. e.g. |
| 99 | |
| 100 | .. code:: python |
| 101 | |
| 102 | # WORKSPACE |
| 103 | |
| 104 | load( |
| 105 | "@pigweed//pw_env_setup/bazel/cipd_setup:cipd_rules.bzl", |
| 106 | "cipd_client_repository", |
| 107 | "cipd_repository", |
| 108 | ) |
| 109 | |
| 110 | # Must be called before cipd_repository |
| 111 | cipd_client_repository() |
| 112 | |
| 113 | cipd_repository( |
| 114 | name = "bloaty", |
| 115 | path = "pigweed/third_party/bloaty-embedded/${os=linux,mac}-${arch=amd64}", |
| 116 | tag = "git_revision:2d87d204057b419f5290f8d38b61b9c2c5b4fb52-2", |
| 117 | ) |
| 118 | |
| 119 | From here it is possible to get access to the Bloaty binaries using the |
| 120 | following command. |
| 121 | |
| 122 | .. code:: sh |
| 123 | |
| 124 | bazel run @bloaty//:bloaty -- --help |
| 125 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 126 | User-Friendliness |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 127 | ----------------- |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 128 | |
| 129 | You may wish to allow sourcing `bootstrap.sh` from a different directory. In |
| 130 | that case you'll need the following at the top of `bootstrap.sh`. |
| 131 | |
| 132 | .. code-block:: bash |
| 133 | |
| 134 | _python_abspath () { |
| 135 | python -c "import os.path; print(os.path.abspath('$@'))" |
| 136 | } |
| 137 | |
| 138 | # Use this code from Pigweed's bootstrap to find the path to this script when |
| 139 | # sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in |
| 140 | # presubmit tests with strange setups, and can be omitted if you're not using |
| 141 | # Pigweed's automated testing infrastructure. |
| 142 | if test -n "$PW_CHECKOUT_ROOT"; then |
| 143 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")" |
| 144 | unset PW_CHECKOUT_ROOT |
| 145 | # Shell: bash. |
| 146 | elif test -n "$BASH"; then |
| 147 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")" |
| 148 | # Shell: zsh. |
| 149 | elif test -n "$ZSH_NAME"; then |
| 150 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")" |
| 151 | # Shell: dash. |
| 152 | elif test ${0##*/} = dash; then |
| 153 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \ |
| 154 | "$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")" |
| 155 | # If everything else fails, try $0. It could work. |
| 156 | else |
| 157 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")" |
| 158 | fi |
| 159 | |
| 160 | You may also wish to check if the user is attempting to execute `bootstrap.sh` |
| 161 | instead of sourcing it. Executing `bootstrap.sh` would download everything |
| 162 | required for the environment, but cannot modify the environment of the parent |
| 163 | process. To check for this add the following. |
| 164 | |
| 165 | .. code-block:: bash |
| 166 | |
| 167 | # Check if this file is being executed or sourced. |
| 168 | _pw_sourced=0 |
| 169 | # If not running in Pigweed's automated testing infrastructure the |
| 170 | # SWARMING_BOT_ID check is unnecessary. |
| 171 | if [ -n "$SWARMING_BOT_ID" ]; then |
| 172 | # If set we're running on swarming and don't need this check. |
| 173 | _pw_sourced=1 |
| 174 | elif [ -n "$ZSH_EVAL_CONTEXT" ]; then |
| 175 | case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac |
| 176 | elif [ -n "$KSH_VERSION" ]; then |
| 177 | [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \ |
| 178 | "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \ |
| 179 | && _pw_sourced=1 |
| 180 | elif [ -n "$BASH_VERSION" ]; then |
| 181 | (return 0 2>/dev/null) && _pw_sourced=1 |
| 182 | else # All other shells: examine $0 for known shell binary filenames |
| 183 | # Detects `sh` and `dash`; add additional shell filenames as needed. |
| 184 | case ${0##*/} in sh|dash) _pw_sourced=1;; esac |
| 185 | fi |
| 186 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 187 | _pw_eval_sourced "$_pw_sourced" |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 188 | |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 189 | Downstream Projects Using Different Packages |
| 190 | ******************************************** |
| 191 | |
| 192 | Projects depending on Pigweed but using additional or different packages should |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 193 | copy the Pigweed `sample project`'s ``bootstrap.sh`` and ``config.json`` and |
| 194 | update the call to ``pw_bootstrap``. Search for "downstream" for other places |
| 195 | that may require changes, like setting the ``PW_ROOT`` and ``PW_PROJECT_ROOT`` |
| 196 | environment variables. Explanations of parts of ``config.json`` are described |
| 197 | here. |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 198 | |
Rob Mohr | 55bb0ad | 2021-05-22 10:59:52 -0700 | [diff] [blame] | 199 | .. _sample project: https://pigweed.googlesource.com/pigweed/sample_project/+/HEAD |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 200 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 201 | ``cipd_package_files`` |
Rob Mohr | 09dc0a3 | 2021-05-19 11:48:30 -0700 | [diff] [blame] | 202 | CIPD package file. JSON file consisting of a list of dictionaries with "path", |
| 203 | "platforms", and "tags" keys. An example is below. |
| 204 | |
| 205 | .. code-block:: json |
| 206 | |
| 207 | { |
| 208 | "path": "infra/3pp/tools/go/${platform}", |
| 209 | "platforms": [ |
| 210 | "linux-amd64", |
| 211 | "linux-arm64", |
| 212 | "mac-amd64", |
| 213 | "windows-amd64" |
| 214 | ], |
| 215 | "tags": [ |
| 216 | "version:2@1.16.3" |
| 217 | ] |
Ewout van Bekkum | 653792b | 2021-05-20 17:35:25 -0700 | [diff] [blame] | 218 | } |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 219 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 220 | ``virtualenv.gn_targets`` |
| 221 | Target for installing Python packages. Downstream projects will need to |
| 222 | create targets to install their packages or only use Pigweed Python packages. |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 223 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 224 | ``virtualenv.gn_root`` |
| 225 | The root directory of your GN build tree, relative to ``PW_PROJECT_ROOT``. |
| 226 | This is the directory your project's ``.gn`` file is located in. If you're |
| 227 | only installing Pigweed Python packages, use the location of the Pigweed |
| 228 | submodule. |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 229 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 230 | An example of a config file is below. |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 231 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 232 | .. code-block:: json |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 233 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 234 | { |
| 235 | "cipd_package_files": [ |
| 236 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json", |
| 237 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json" |
Rob Mohr | 0464560 | 2021-03-03 11:26:05 -0800 | [diff] [blame] | 238 | "tools/myprojectname.json" |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 239 | ], |
| 240 | "virtualenv": { |
| 241 | "gn_root": ".", |
| 242 | "gn_targets": [ |
| 243 | ":python.install", |
| 244 | ] |
| 245 | } |
| 246 | } |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 247 | |
Rob Mohr | 0464560 | 2021-03-03 11:26:05 -0800 | [diff] [blame] | 248 | In case the CIPD packages need to be referenced from other scripts, variables |
| 249 | like ``PW_${BASENAME}_CIPD_INSTALL_DIR`` point to the CIPD install directories, |
| 250 | where ``${BASENAME}`` is "PIGWEED" for |
| 251 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json" and "LUCI" for |
| 252 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json". This example would |
| 253 | set the following environment variables. |
| 254 | |
| 255 | - ``PW_LUCI_CIPD_INSTALL_DIR`` |
| 256 | - ``PW_MYPROJECTNAME_CIPD_INSTALL_DIR`` |
| 257 | - ``PW_PIGWEED_CIPD_INSTALL_DIR`` |
| 258 | |
Rob Mohr | 9891ced | 2021-01-13 07:49:28 -0800 | [diff] [blame] | 259 | Environment Variables |
| 260 | ********************* |
| 261 | The following environment variables affect env setup behavior. Most users will |
| 262 | never need to set these. |
| 263 | |
| 264 | ``CIPD_CACHE_DIR`` |
| 265 | Location of CIPD cache dir. Defaults to ``$HOME/.cipd-cache-dir``. |
| 266 | |
| 267 | ``PW_ACTIVATE_SKIP_CHECKS`` |
| 268 | If set, skip running ``pw doctor`` at end of bootstrap/activate. Intended to |
| 269 | be used by automated tools but not interactively. |
| 270 | |
| 271 | ``PW_BOOTSTRAP_PYTHON`` |
| 272 | Python executable to be used, for example "python2" or "python3". Defaults to |
| 273 | "python". |
| 274 | |
| 275 | ``PW_ENVIRONMENT_ROOT`` |
| 276 | Location to which packages are installed. Defaults to ``.environment`` folder |
| 277 | within the checkout root. |
| 278 | |
| 279 | ``PW_ENVSETUP_DISABLE_SPINNER`` |
| 280 | Disable the spinner during env setup. Intended to be used when the output is |
| 281 | being redirected to a log. |
| 282 | |
| 283 | ``PW_ENVSETUP_QUIET`` |
| 284 | Disables all non-error output. |
Rob Mohr | e9d5aac | 2021-01-05 13:12:10 -0800 | [diff] [blame] | 285 | |
Rob Mohr | 1d2ffdf | 2021-05-12 14:44:45 -0700 | [diff] [blame] | 286 | Non-Shell Environments |
| 287 | ********************** |
| 288 | If using this outside of bash—for example directly from an IDE or CI |
| 289 | system—users can process the ``actions.json`` file that's generated in the |
| 290 | environment directory. It lists variables to set, clear, and modify. An |
| 291 | example ``actions.json`` is shown below. The "append" and "prepend" actions |
| 292 | are listed in the order they should be applied, so the |
| 293 | ``<pigweed-root>/out/host/host_tools`` entry should be at the beginning of |
| 294 | ``PATH`` and not in the middle somewhere. |
| 295 | |
| 296 | .. code-block:: json |
| 297 | |
| 298 | { |
| 299 | "modify": { |
| 300 | "PATH": { |
| 301 | "append": [], |
| 302 | "prepend": [ |
| 303 | "<pigweed-root>/.environment/cipd", |
| 304 | "<pigweed-root>/.environment/cipd/pigweed", |
| 305 | "<pigweed-root>/.environment/cipd/pigweed/bin", |
| 306 | "<pigweed-root>/.environment/cipd/luci", |
| 307 | "<pigweed-root>/.environment/cipd/luci/bin", |
| 308 | "<pigweed-root>/.environment/pigweed-venv/bin", |
| 309 | "<pigweed-root>/out/host/host_tools" |
| 310 | ], |
| 311 | "remove": [] |
| 312 | } |
| 313 | }, |
| 314 | "set": { |
| 315 | "PW_PROJECT_ROOT": "<pigweed-root>", |
| 316 | "PW_ROOT": "<pigweed-root>", |
| 317 | "_PW_ACTUAL_ENVIRONMENT_ROOT": "<pigweed-root>/.environment", |
| 318 | "PW_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd", |
| 319 | "CIPD_CACHE_DIR": "/usr/local/google/home/mohrr/.cipd-cache-dir", |
| 320 | "PW_PIGWEED_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd/pigweed", |
| 321 | "PW_LUCI_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd/luci", |
| 322 | "VIRTUAL_ENV": "<pigweed-root>/.environment/pigweed-venv", |
| 323 | "PYTHONHOME": null, |
| 324 | "__PYVENV_LAUNCHER__": null |
| 325 | } |
| 326 | } |
| 327 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 328 | Implementation |
| 329 | ************** |
| 330 | |
| 331 | The environment is set up by installing CIPD and Python packages in |
| 332 | ``PW_ENVIRONMENT_ROOT`` or ``<checkout>/.environment``, and saving modifications |
| 333 | to environment variables in setup scripts in those directories. To support |
| 334 | multiple operating systems this is done in an operating system-agnostic manner |
| 335 | and then written into operating system-specific files to be sourced now and in |
| 336 | the future when running ``activate.sh`` instead of ``bootstrap.sh``. In the |
| 337 | future these could be extended to C shell and PowerShell. A logical mapping of |
| 338 | high-level commands to system-specific initialization files is shown below. |
| 339 | |
| 340 | .. image:: doc_resources/pw_env_setup_output.png |
| 341 | :alt: Mapping of high-level commands to system-specific commands. |
| 342 | :align: left |