Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_env_setup: |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 2 | |
| 3 | ------------ |
| 4 | pw_env_setup |
| 5 | ------------ |
| 6 | A classic problem in the embedded space is reducing the time from git clone |
| 7 | to having a binary executing on a device. The issue is that an entire suite |
| 8 | of tools is needed for non-trivial production embedded projects. For example: |
| 9 | |
| 10 | - A C++ compiler for your target device, and also for your host |
| 11 | - A build system or three; for example, GN, Ninja, CMake, Bazel |
| 12 | - A code formatting program like clang-format |
| 13 | - A debugger like OpenOCD to flash and debug your embedded device |
| 14 | - A known Python version with known modules installed for scripting |
| 15 | - A Go compiler for the Go-based command line tools |
| 16 | |
| 17 | ...and so on |
| 18 | |
| 19 | In the server space, container solutions like Docker or Podman solve this; |
| 20 | however, in our experience container solutions are a mixed bag for embedded |
| 21 | systems development where one frequently needs access to native system |
| 22 | resources like USB devices, or must operate on Windows. |
| 23 | |
Armando Montanez | 0054a9b | 2020-03-13 13:06:24 -0700 | [diff] [blame] | 24 | ``pw_env_setup`` is our compromise solution for this problem that works on Mac, |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 25 | Windows, and Linux. It leverages the Chrome packaging system `CIPD`_ to |
| 26 | bootstrap a Python installation, which in turn inflates a virtual |
| 27 | environment. The tooling is installed into your workspace, and makes no |
| 28 | changes to your system. This tooling is designed to be reused by any |
| 29 | project. |
| 30 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 31 | .. _CIPD: https://github.com/luci/luci-go/tree/master/cipd |
| 32 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 33 | Users interact with ``pw_env_setup`` with two commands: ``. bootstrap.sh`` and |
| 34 | ``. activate.sh``. The bootstrap command always pulls down the current versions |
| 35 | of CIPD packages and sets up the Python virtual environment. The activate |
| 36 | command reinitializes a previously configured environment, and if none is found, |
| 37 | runs bootstrap. |
| 38 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 39 | .. note:: |
| 40 | On Windows the scripts used to set up the environment are ``bootstrap.bat`` |
| 41 | and ``activate.bat``. For simplicity they will be referred to with the ``.sh`` |
| 42 | endings unless the distinction is relevant. |
| 43 | |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 44 | .. warning:: |
Armando Montanez | 0054a9b | 2020-03-13 13:06:24 -0700 | [diff] [blame] | 45 | At this time ``pw_env_setup`` works for us, but isn’t well tested. We don’t |
Rob Mohr | d45e234 | 2020-03-03 15:02:18 -0800 | [diff] [blame] | 46 | suggest relying on it just yet. However, we are interested in experience |
| 47 | reports; if you give it a try, please `send us a note`_ about your |
| 48 | experience. |
| 49 | |
| 50 | .. _send us a note: pigweed@googlegroups.com |
Rob Mohr | d596a53 | 2020-04-03 08:59:12 -0700 | [diff] [blame] | 51 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 52 | ================================== |
| 53 | Using pw_env_setup in your project |
| 54 | ================================== |
| 55 | |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 56 | Downstream Projects Using Pigweed's Packages |
| 57 | ******************************************** |
Rob Mohr | d596a53 | 2020-04-03 08:59:12 -0700 | [diff] [blame] | 58 | |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 59 | Projects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's |
| 60 | dependencies or their own dependencies. Projects that only want to use Pigweed's |
| 61 | dependencies without modifying them can just source Pigweed's ``bootstrap.sh`` |
| 62 | and ``activate.sh`` scripts. |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 63 | |
| 64 | An example of what your project's `bootstrap.sh` could look like is below. This |
| 65 | assumes `bootstrap.sh` is at the top level of your repository. |
| 66 | |
| 67 | .. code-block:: bash |
| 68 | |
| 69 | # Do not include a "#!" line, this must be sourced and not executed. |
| 70 | |
| 71 | # This assumes the user is sourcing this file from it's parent directory. See |
| 72 | # below for a more flexible way to handle this. |
| 73 | PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh" |
| 74 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 75 | export PW_PROJECT_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")" |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 76 | |
| 77 | # You may wish to check if the user is attempting to execute this script |
| 78 | # instead of sourcing it. See below for an example of how to handle that |
| 79 | # situation. |
| 80 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 81 | # Source Pigweed's bootstrap utility script. |
| 82 | # Using '.' instead of 'source' for POSIX compatibility. Since users don't use |
| 83 | # dash directly, using 'source' in most documentation so users don't get |
| 84 | # confused and try to `./bootstrap.sh`. |
| 85 | . "$PW_PROJECT_ROOT/third_party/pigweed/pw_env_setup/util.sh" |
| 86 | |
| 87 | pw_check_root "$PW_ROOT" |
| 88 | _PW_ACTUAL_ENVIRONMENT_ROOT="$(pw_get_env_root)" |
| 89 | export _PW_ACTUAL_ENVIRONMENT_ROOT |
| 90 | SETUP_SH="$_PW_ACTUAL_ENVIRONMENT_ROOT/activate.sh" |
| 91 | pw_bootstrap --args... # See below for details about args. |
| 92 | pw_finalize bootstrap "$SETUP_SH" |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 93 | |
| 94 | User-Friendliness |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 95 | ----------------- |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 96 | |
| 97 | You may wish to allow sourcing `bootstrap.sh` from a different directory. In |
| 98 | that case you'll need the following at the top of `bootstrap.sh`. |
| 99 | |
| 100 | .. code-block:: bash |
| 101 | |
| 102 | _python_abspath () { |
| 103 | python -c "import os.path; print(os.path.abspath('$@'))" |
| 104 | } |
| 105 | |
| 106 | # Use this code from Pigweed's bootstrap to find the path to this script when |
| 107 | # sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in |
| 108 | # presubmit tests with strange setups, and can be omitted if you're not using |
| 109 | # Pigweed's automated testing infrastructure. |
| 110 | if test -n "$PW_CHECKOUT_ROOT"; then |
| 111 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")" |
| 112 | unset PW_CHECKOUT_ROOT |
| 113 | # Shell: bash. |
| 114 | elif test -n "$BASH"; then |
| 115 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")" |
| 116 | # Shell: zsh. |
| 117 | elif test -n "$ZSH_NAME"; then |
| 118 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")" |
| 119 | # Shell: dash. |
| 120 | elif test ${0##*/} = dash; then |
| 121 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \ |
| 122 | "$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")" |
| 123 | # If everything else fails, try $0. It could work. |
| 124 | else |
| 125 | PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")" |
| 126 | fi |
| 127 | |
| 128 | You may also wish to check if the user is attempting to execute `bootstrap.sh` |
| 129 | instead of sourcing it. Executing `bootstrap.sh` would download everything |
| 130 | required for the environment, but cannot modify the environment of the parent |
| 131 | process. To check for this add the following. |
| 132 | |
| 133 | .. code-block:: bash |
| 134 | |
| 135 | # Check if this file is being executed or sourced. |
| 136 | _pw_sourced=0 |
| 137 | # If not running in Pigweed's automated testing infrastructure the |
| 138 | # SWARMING_BOT_ID check is unnecessary. |
| 139 | if [ -n "$SWARMING_BOT_ID" ]; then |
| 140 | # If set we're running on swarming and don't need this check. |
| 141 | _pw_sourced=1 |
| 142 | elif [ -n "$ZSH_EVAL_CONTEXT" ]; then |
| 143 | case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac |
| 144 | elif [ -n "$KSH_VERSION" ]; then |
| 145 | [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \ |
| 146 | "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \ |
| 147 | && _pw_sourced=1 |
| 148 | elif [ -n "$BASH_VERSION" ]; then |
| 149 | (return 0 2>/dev/null) && _pw_sourced=1 |
| 150 | else # All other shells: examine $0 for known shell binary filenames |
| 151 | # Detects `sh` and `dash`; add additional shell filenames as needed. |
| 152 | case ${0##*/} in sh|dash) _pw_sourced=1;; esac |
| 153 | fi |
| 154 | |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 155 | _pw_eval_sourced "$_pw_sourced" |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 156 | |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 157 | Downstream Projects Using Different Packages |
| 158 | ******************************************** |
| 159 | |
| 160 | Projects depending on Pigweed but using additional or different packages should |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 161 | copy the Pigweed `sample project`'s ``bootstrap.sh`` and ``config.json`` and |
| 162 | update the call to ``pw_bootstrap``. Search for "downstream" for other places |
| 163 | that may require changes, like setting the ``PW_ROOT`` and ``PW_PROJECT_ROOT`` |
| 164 | environment variables. Explanations of parts of ``config.json`` are described |
| 165 | here. |
Rob Mohr | fcf6037 | 2020-11-04 11:41:36 -0800 | [diff] [blame] | 166 | |
| 167 | .. _sample project: https://pigweed.googlesource.com/pigweed/sample_project/+/master |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 168 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 169 | ``cipd_package_files`` |
Rob Mohr | 09dc0a3 | 2021-05-19 11:48:30 -0700 | [diff] [blame^] | 170 | CIPD package file. JSON file consisting of a list of dictionaries with "path", |
| 171 | "platforms", and "tags" keys. An example is below. |
| 172 | |
| 173 | .. code-block:: json |
| 174 | |
| 175 | { |
| 176 | "path": "infra/3pp/tools/go/${platform}", |
| 177 | "platforms": [ |
| 178 | "linux-amd64", |
| 179 | "linux-arm64", |
| 180 | "mac-amd64", |
| 181 | "windows-amd64" |
| 182 | ], |
| 183 | "tags": [ |
| 184 | "version:2@1.16.3" |
| 185 | ] |
| 186 | }, |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 187 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 188 | ``virtualenv.gn_targets`` |
| 189 | Target for installing Python packages. Downstream projects will need to |
| 190 | create targets to install their packages or only use Pigweed Python packages. |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 191 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 192 | ``virtualenv.gn_root`` |
| 193 | The root directory of your GN build tree, relative to ``PW_PROJECT_ROOT``. |
| 194 | This is the directory your project's ``.gn`` file is located in. If you're |
| 195 | only installing Pigweed Python packages, use the location of the Pigweed |
| 196 | submodule. |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 197 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 198 | An example of a config file is below. |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 199 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 200 | .. code-block:: json |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 201 | |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 202 | { |
| 203 | "cipd_package_files": [ |
| 204 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json", |
| 205 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json" |
Rob Mohr | 0464560 | 2021-03-03 11:26:05 -0800 | [diff] [blame] | 206 | "tools/myprojectname.json" |
Rob Mohr | cccd63c | 2021-02-26 12:03:07 -0800 | [diff] [blame] | 207 | ], |
| 208 | "virtualenv": { |
| 209 | "gn_root": ".", |
| 210 | "gn_targets": [ |
| 211 | ":python.install", |
| 212 | ] |
| 213 | } |
| 214 | } |
Rob Mohr | 895d3df | 2020-10-14 08:42:39 -0700 | [diff] [blame] | 215 | |
Rob Mohr | 0464560 | 2021-03-03 11:26:05 -0800 | [diff] [blame] | 216 | In case the CIPD packages need to be referenced from other scripts, variables |
| 217 | like ``PW_${BASENAME}_CIPD_INSTALL_DIR`` point to the CIPD install directories, |
| 218 | where ``${BASENAME}`` is "PIGWEED" for |
| 219 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json" and "LUCI" for |
| 220 | "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json". This example would |
| 221 | set the following environment variables. |
| 222 | |
| 223 | - ``PW_LUCI_CIPD_INSTALL_DIR`` |
| 224 | - ``PW_MYPROJECTNAME_CIPD_INSTALL_DIR`` |
| 225 | - ``PW_PIGWEED_CIPD_INSTALL_DIR`` |
| 226 | |
Rob Mohr | 9891ced | 2021-01-13 07:49:28 -0800 | [diff] [blame] | 227 | Environment Variables |
| 228 | ********************* |
| 229 | The following environment variables affect env setup behavior. Most users will |
| 230 | never need to set these. |
| 231 | |
| 232 | ``CIPD_CACHE_DIR`` |
| 233 | Location of CIPD cache dir. Defaults to ``$HOME/.cipd-cache-dir``. |
| 234 | |
| 235 | ``PW_ACTIVATE_SKIP_CHECKS`` |
| 236 | If set, skip running ``pw doctor`` at end of bootstrap/activate. Intended to |
| 237 | be used by automated tools but not interactively. |
| 238 | |
| 239 | ``PW_BOOTSTRAP_PYTHON`` |
| 240 | Python executable to be used, for example "python2" or "python3". Defaults to |
| 241 | "python". |
| 242 | |
| 243 | ``PW_ENVIRONMENT_ROOT`` |
| 244 | Location to which packages are installed. Defaults to ``.environment`` folder |
| 245 | within the checkout root. |
| 246 | |
| 247 | ``PW_ENVSETUP_DISABLE_SPINNER`` |
| 248 | Disable the spinner during env setup. Intended to be used when the output is |
| 249 | being redirected to a log. |
| 250 | |
| 251 | ``PW_ENVSETUP_QUIET`` |
| 252 | Disables all non-error output. |
Rob Mohr | e9d5aac | 2021-01-05 13:12:10 -0800 | [diff] [blame] | 253 | |
Rob Mohr | 1d2ffdf | 2021-05-12 14:44:45 -0700 | [diff] [blame] | 254 | Non-Shell Environments |
| 255 | ********************** |
| 256 | If using this outside of bash—for example directly from an IDE or CI |
| 257 | system—users can process the ``actions.json`` file that's generated in the |
| 258 | environment directory. It lists variables to set, clear, and modify. An |
| 259 | example ``actions.json`` is shown below. The "append" and "prepend" actions |
| 260 | are listed in the order they should be applied, so the |
| 261 | ``<pigweed-root>/out/host/host_tools`` entry should be at the beginning of |
| 262 | ``PATH`` and not in the middle somewhere. |
| 263 | |
| 264 | .. code-block:: json |
| 265 | |
| 266 | { |
| 267 | "modify": { |
| 268 | "PATH": { |
| 269 | "append": [], |
| 270 | "prepend": [ |
| 271 | "<pigweed-root>/.environment/cipd", |
| 272 | "<pigweed-root>/.environment/cipd/pigweed", |
| 273 | "<pigweed-root>/.environment/cipd/pigweed/bin", |
| 274 | "<pigweed-root>/.environment/cipd/luci", |
| 275 | "<pigweed-root>/.environment/cipd/luci/bin", |
| 276 | "<pigweed-root>/.environment/pigweed-venv/bin", |
| 277 | "<pigweed-root>/out/host/host_tools" |
| 278 | ], |
| 279 | "remove": [] |
| 280 | } |
| 281 | }, |
| 282 | "set": { |
| 283 | "PW_PROJECT_ROOT": "<pigweed-root>", |
| 284 | "PW_ROOT": "<pigweed-root>", |
| 285 | "_PW_ACTUAL_ENVIRONMENT_ROOT": "<pigweed-root>/.environment", |
| 286 | "PW_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd", |
| 287 | "CIPD_CACHE_DIR": "/usr/local/google/home/mohrr/.cipd-cache-dir", |
| 288 | "PW_PIGWEED_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd/pigweed", |
| 289 | "PW_LUCI_CIPD_INSTALL_DIR": "<pigweed-root>/.environment/cipd/luci", |
| 290 | "VIRTUAL_ENV": "<pigweed-root>/.environment/pigweed-venv", |
| 291 | "PYTHONHOME": null, |
| 292 | "__PYVENV_LAUNCHER__": null |
| 293 | } |
| 294 | } |
| 295 | |
Rob Mohr | 43d3611 | 2020-09-09 13:38:13 -0700 | [diff] [blame] | 296 | Implementation |
| 297 | ************** |
| 298 | |
| 299 | The environment is set up by installing CIPD and Python packages in |
| 300 | ``PW_ENVIRONMENT_ROOT`` or ``<checkout>/.environment``, and saving modifications |
| 301 | to environment variables in setup scripts in those directories. To support |
| 302 | multiple operating systems this is done in an operating system-agnostic manner |
| 303 | and then written into operating system-specific files to be sourced now and in |
| 304 | the future when running ``activate.sh`` instead of ``bootstrap.sh``. In the |
| 305 | future these could be extended to C shell and PowerShell. A logical mapping of |
| 306 | high-level commands to system-specific initialization files is shown below. |
| 307 | |
| 308 | .. image:: doc_resources/pw_env_setup_output.png |
| 309 | :alt: Mapping of high-level commands to system-specific commands. |
| 310 | :align: left |