Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 1 | .. _module-pw_presubmit: |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 2 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 3 | ============ |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 4 | pw_presubmit |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 5 | ============ |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 6 | The presubmit module provides Python tools for running presubmit checks and |
| 7 | checking and fixing code format. It also includes the presubmit check script for |
| 8 | the Pigweed repository, ``pigweed_presubmit.py``. |
| 9 | |
Keir Mierle | 086ef1c | 2020-03-19 02:03:51 -0700 | [diff] [blame] | 10 | Presubmit checks are essential tools, but they take work to set up, and |
| 11 | projects don’t always get around to it. The ``pw_presubmit`` module provides |
| 12 | tools for setting up high quality presubmit checks for any project. We use this |
| 13 | framework to run Pigweed’s presubmit on our workstations and in our automated |
| 14 | building tools. |
| 15 | |
| 16 | The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a |
| 17 | unified interface for automatically formatting code in a variety of languages. |
| 18 | With ``pw format``, you can format C, C++, Python, GN, and Go code according to |
| 19 | configurations defined by your project. ``pw format`` leverages existing tools |
| 20 | like ``clang-format``, and it’s simple to add support for new languages. |
| 21 | |
Alexei Frolov | 133eb7d | 2020-09-08 15:58:12 -0700 | [diff] [blame] | 22 | .. image:: docs/pw_presubmit_demo.gif |
Keir Mierle | 086ef1c | 2020-03-19 02:03:51 -0700 | [diff] [blame] | 23 | :alt: ``pw format`` demo |
| 24 | :align: left |
| 25 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 26 | The ``pw_presubmit`` package includes presubmit checks that can be used with any |
| 27 | project. These checks include: |
| 28 | |
| 29 | * Check code format of several languages including C, C++, and Python |
| 30 | * Initialize a Python environment |
| 31 | * Run all Python tests |
| 32 | * Run pylint |
| 33 | * Run mypy |
| 34 | * Ensure source files are included in the GN and Bazel builds |
| 35 | * Build and run all tests with GN |
| 36 | * Build and run all tests with Bazel |
| 37 | * Ensure all header files contain ``#pragma once`` |
| 38 | |
| 39 | ------------- |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 40 | Compatibility |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 41 | ------------- |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 42 | Python 3 |
| 43 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 44 | ------------------------------------------- |
| 45 | Creating a presubmit check for your project |
| 46 | ------------------------------------------- |
| 47 | Creating a presubmit check for a project using ``pw_presubmit`` is simple, but |
| 48 | requires some customization. Projects must define their own presubmit check |
| 49 | Python script that uses the ``pw_presubmit`` package. |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 50 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 51 | A project's presubmit script can be registered as a |
Wyatt Hepler | f9fb90f | 2020-09-30 18:59:33 -0700 | [diff] [blame] | 52 | :ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 53 | presubmit``. |
| 54 | |
| 55 | Setting up the command-line interface |
| 56 | ------------------------------------- |
| 57 | The ``pw_presubmit.cli`` module sets up the command-line interface for a |
| 58 | presubmit script. This defines a standard set of arguments for invoking |
| 59 | presubmit checks. Its use is optional, but recommended. |
| 60 | |
| 61 | pw_presubmit.cli |
| 62 | ~~~~~~~~~~~~~~~~ |
| 63 | .. automodule:: pw_presubmit.cli |
| 64 | :members: add_arguments, run |
| 65 | |
| 66 | Presubmit checks |
| 67 | ---------------- |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 68 | A presubmit check is defined as a function or other callable. The function must |
| 69 | accept one argument: a ``PresubmitContext``, which provides the paths on which |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 70 | to run. Presubmit checks communicate failure by raising an exception. |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 71 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 72 | Presubmit checks may use the ``filter_paths`` decorator to automatically filter |
| 73 | the paths list for file types they care about. |
| 74 | |
| 75 | Either of these functions could be used as presubmit checks: |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 76 | |
| 77 | .. code-block:: python |
| 78 | |
| 79 | @pw_presubmit.filter_paths(endswith='.py') |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 80 | def file_contains_ni(ctx: PresubmitContext): |
| 81 | for path in ctx.paths: |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 82 | with open(path) as file: |
| 83 | contents = file.read() |
| 84 | if 'ni' not in contents and 'nee' not in contents: |
| 85 | raise PresumitFailure('Files must say "ni"!', path=path) |
| 86 | |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 87 | def run_the_build(_): |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 88 | subprocess.run(['make', 'release'], check=True) |
| 89 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 90 | Presubmit checks functions are grouped into "programs" -- a named series of |
| 91 | checks. Projects may find it helpful to have programs for different purposes, |
| 92 | such as a quick program for local use and a full program for automated use. The |
| 93 | :ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define |
| 94 | ``quick`` and ``full`` programs. |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 95 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 96 | pw_presubmit |
| 97 | ~~~~~~~~~~~~ |
| 98 | .. automodule:: pw_presubmit |
| 99 | :members: filter_paths, call, PresubmitFailure, Programs |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 100 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 101 | .. _example-script: |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 102 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 103 | Example |
| 104 | ------- |
| 105 | A simple example presubmit check script follows. This can be copied-and-pasted |
| 106 | to serve as a starting point for a project's presubmit check script. |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 107 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 108 | See ``pigweed_presubmit.py`` for a more complex presubmit check script example. |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 109 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 110 | .. code-block:: python |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 111 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 112 | """Example presubmit check script.""" |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 113 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 114 | import argparse |
| 115 | import logging |
| 116 | import os |
| 117 | from pathlib import Path |
| 118 | import re |
| 119 | import sys |
| 120 | from typing import List, Pattern |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 121 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 122 | try: |
| 123 | import pw_cli.log |
| 124 | except ImportError: |
| 125 | print('ERROR: Activate the environment before running presubmits!', |
| 126 | file=sys.stderr) |
| 127 | sys.exit(2) |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 128 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 129 | import pw_presubmit |
| 130 | from pw_presubmit import build, cli, environment, format_code, git_repo |
| 131 | from pw_presubmit import python_checks, filter_paths, PresubmitContext |
| 132 | from pw_presubmit.install_hook import install_hook |
Wyatt Hepler | ee3e02f | 2019-12-05 10:52:31 -0800 | [diff] [blame] | 133 | |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 134 | # Set up variables for key project paths. |
| 135 | PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT']) |
| 136 | PIGWEED_ROOT = PROJECT_ROOT / 'pigweed' |
| 137 | |
| 138 | # |
| 139 | # Initialization |
| 140 | # |
| 141 | def init_cipd(ctx: PresubmitContext): |
| 142 | environment.init_cipd(PIGWEED_ROOT, ctx.output_dir) |
| 143 | |
| 144 | |
| 145 | def init_virtualenv(ctx: PresubmitContext): |
| 146 | environment.init_virtualenv(PIGWEED_ROOT, |
| 147 | ctx.output_dir, |
| 148 | setup_py_roots=[PROJECT_ROOT]) |
| 149 | |
| 150 | |
| 151 | # Rerun the build if files with these extensions change. |
| 152 | _BUILD_EXTENSIONS = frozenset( |
| 153 | ['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions]) |
| 154 | |
| 155 | |
| 156 | # |
| 157 | # Presubmit checks |
| 158 | # |
| 159 | def release_build(ctx: PresubmitContext): |
| 160 | build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release') |
| 161 | build.ninja(ctx.output_dir) |
| 162 | |
| 163 | |
| 164 | def host_tests(ctx: PresubmitContext): |
| 165 | build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true') |
| 166 | build.ninja(ctx.output_dir) |
| 167 | |
| 168 | |
| 169 | # Avoid running some checks on certain paths. |
| 170 | PATH_EXCLUSIONS = ( |
| 171 | re.compile(r'^external/'), |
| 172 | re.compile(r'^vendor/'), |
| 173 | ) |
| 174 | |
| 175 | |
| 176 | # Use the upstream pragma_once check, but apply a different set of path |
| 177 | # filters with @filter_paths. |
| 178 | @filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS) |
| 179 | def pragma_once(ctx: PresubmitContext): |
| 180 | pw_presubmit.pragma_once(ctx) |
| 181 | |
| 182 | |
| 183 | # |
| 184 | # Presubmit check programs |
| 185 | # |
| 186 | QUICK = ( |
| 187 | # Initialize an environment for running presubmit checks. |
| 188 | init_cipd, |
| 189 | init_virtualenv, |
| 190 | # List some presubmit checks to run |
| 191 | pragma_once, |
| 192 | host_tests, |
| 193 | # Use the upstream formatting checks, with custom path filters applied. |
| 194 | format_code.presubmit_checks(exclude=PATH_EXCLUSIONS), |
| 195 | ) |
| 196 | |
| 197 | FULL = ( |
| 198 | QUICK, # Add all checks from the 'quick' program |
| 199 | release_build, |
| 200 | # Use the upstream Python checks, with custom path filters applied. |
| 201 | python_checks.all_checks(exclude=PATH_EXCLUSIONS), |
| 202 | ) |
| 203 | |
| 204 | PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL) |
| 205 | |
| 206 | |
| 207 | def run(install: bool, **presubmit_args) -> int: |
| 208 | """Process the --install argument then invoke pw_presubmit.""" |
| 209 | |
| 210 | # Install the presubmit Git pre-push hook, if requested. |
| 211 | if install: |
| 212 | install_hook(__file__, 'pre-push', ['--base', 'HEAD~'], |
| 213 | git_repo.root()) |
| 214 | return 0 |
| 215 | |
| 216 | return cli.run(root=PROJECT_ROOT, **presubmit_args) |
| 217 | |
| 218 | |
| 219 | def main() -> int: |
| 220 | """Run the presubmit checks for this repository.""" |
| 221 | parser = argparse.ArgumentParser(description=__doc__) |
| 222 | cli.add_arguments(parser, PROGRAMS, 'quick') |
| 223 | |
| 224 | # Define an option for installing a Git pre-push hook for this script. |
| 225 | parser.add_argument( |
| 226 | '--install', |
| 227 | action='store_true', |
| 228 | help='Install the presubmit as a Git pre-push hook and exit.') |
| 229 | |
| 230 | return run(**vars(parser.parse_args())) |
| 231 | |
| 232 | if __name__ == '__main__': |
| 233 | pw_cli.log.install(logging.INFO) |
| 234 | sys.exit(main()) |
| 235 | |
| 236 | --------------------- |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 237 | Code formatting tools |
Wyatt Hepler | 62525fd | 2020-06-09 09:23:49 -0700 | [diff] [blame] | 238 | --------------------- |
Wyatt Hepler | 5a4dc59 | 2020-04-29 15:56:01 -0700 | [diff] [blame] | 239 | The ``pw_presubmit.format_code`` module formats supported source files using |
| 240 | external code format tools. The file ``format_code.py`` can be invoked directly |
| 241 | from the command line or from ``pw`` as ``pw format``. |