Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`compileall` --- Byte-compile Python libraries |
| 2 | =================================================== |
| 3 | |
| 4 | .. module:: compileall |
| 5 | :synopsis: Tools for byte-compiling all Python source files in a directory tree. |
| 6 | |
Georg Brandl | c0a8f8c | 2014-10-02 08:38:39 +0200 | [diff] [blame] | 7 | **Source code:** :source:`Lib/compileall.py` |
| 8 | |
| 9 | -------------- |
| 10 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 11 | This module provides some utility functions to support installing Python |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 12 | libraries. These functions compile Python source files in a directory tree. |
| 13 | This module can be used to create the cached byte-code files at library |
| 14 | installation time, which makes them available for use even by users who don't |
| 15 | have write permission to the library directories. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 16 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 17 | |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 18 | Command-line use |
| 19 | ---------------- |
| 20 | |
| 21 | This module can work as a script (using :program:`python -m compileall`) to |
| 22 | compile Python sources. |
| 23 | |
| 24 | .. program:: compileall |
| 25 | |
Georg Brandl | f5c801f | 2014-03-18 07:44:07 +0100 | [diff] [blame] | 26 | .. cmdoption:: directory ... |
| 27 | file ... |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 28 | |
| 29 | Positional arguments are files to compile or directories that contain |
| 30 | source files, traversed recursively. If no argument is given, behave as if |
| 31 | the command line was ``-l <directories from sys.path>``. |
| 32 | |
| 33 | .. cmdoption:: -l |
| 34 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 35 | Do not recurse into subdirectories, only compile source code files directly |
| 36 | contained in the named or implied directories. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 37 | |
| 38 | .. cmdoption:: -f |
| 39 | |
| 40 | Force rebuild even if timestamps are up-to-date. |
| 41 | |
| 42 | .. cmdoption:: -q |
| 43 | |
Berker Peksag | 6554b86 | 2014-10-15 11:10:57 +0300 | [diff] [blame] | 44 | Do not print the list of files compiled. If passed once, error messages will |
| 45 | still be printed. If passed twice (``-qq``), all output is suppressed. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 46 | |
| 47 | .. cmdoption:: -d destdir |
| 48 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 49 | Directory prepended to the path to each file being compiled. This will |
| 50 | appear in compilation time tracebacks, and is also compiled in to the |
| 51 | byte-code file, where it will be used in tracebacks and other messages in |
| 52 | cases where the source file does not exist at the time the byte-code file is |
| 53 | executed. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 54 | |
| 55 | .. cmdoption:: -x regex |
| 56 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 57 | regex is used to search the full path to each file considered for |
| 58 | compilation, and if the regex produces a match, the file is skipped. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 59 | |
| 60 | .. cmdoption:: -i list |
| 61 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 62 | Read the file ``list`` and add each line that it contains to the list of |
| 63 | files and directories to compile. If ``list`` is ``-``, read lines from |
| 64 | ``stdin``. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 65 | |
| 66 | .. cmdoption:: -b |
| 67 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 68 | Write the byte-code files to their legacy locations and names, which may |
| 69 | overwrite byte-code files created by another version of Python. The default |
| 70 | is to write files to their :pep:`3147` locations and names, which allows |
| 71 | byte-code files from multiple versions of Python to coexist. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 72 | |
Benjamin Peterson | 344ff4a | 2014-08-19 16:13:26 -0500 | [diff] [blame] | 73 | .. cmdoption:: -r |
| 74 | |
| 75 | Control the maximum recursion level for subdirectories. |
| 76 | If this is given, then ``-l`` option will not be taken into account. |
| 77 | :program:`python -m compileall <directory> -r 0` is equivalent to |
| 78 | :program:`python -m compileall <directory> -l`. |
| 79 | |
Brett Cannon | f1a8df0 | 2014-09-12 10:39:48 -0400 | [diff] [blame] | 80 | .. cmdoption:: -j N |
| 81 | |
| 82 | Use *N* workers to compile the files within the given directory. |
| 83 | If ``0`` is used, then the result of :func:`os.cpu_count()` |
| 84 | will be used. |
Benjamin Peterson | 344ff4a | 2014-08-19 16:13:26 -0500 | [diff] [blame] | 85 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 86 | .. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash] |
| 87 | |
| 88 | Control how the generated pycs will be invalidated at runtime. The default |
| 89 | setting, ``timestamp``, means that ``.pyc`` files with the source timestamp |
| 90 | and size embedded will be generated. The ``checked-hash`` and |
| 91 | ``unchecked-hash`` values cause hash-based pycs to be generated. Hash-based |
| 92 | pycs embed a hash of the source file contents rather than a timestamp. See |
| 93 | :ref:`pyc-invalidation` for more information on how Python validates bytecode |
| 94 | cache files at runtime. |
| 95 | |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 96 | .. versionchanged:: 3.2 |
| 97 | Added the ``-i``, ``-b`` and ``-h`` options. |
Éric Araujo | f68fa05 | 2010-12-16 02:10:11 +0000 | [diff] [blame] | 98 | |
Benjamin Peterson | 344ff4a | 2014-08-19 16:13:26 -0500 | [diff] [blame] | 99 | .. versionchanged:: 3.5 |
Yury Selivanov | f03d50c | 2015-09-09 09:32:07 -0400 | [diff] [blame] | 100 | Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option |
Serhiy Storchaka | 2446eab | 2015-09-13 21:09:17 +0300 | [diff] [blame] | 101 | was changed to a multilevel value. ``-b`` will always produce a |
Yury Selivanov | f03d50c | 2015-09-09 09:32:07 -0400 | [diff] [blame] | 102 | byte-code file ending in ``.pyc``, never ``.pyo``. |
Brett Cannon | f299abd | 2015-04-13 14:21:02 -0400 | [diff] [blame] | 103 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 104 | .. versionchanged:: 3.7 |
| 105 | Added the ``--invalidation-mode`` parameter. |
| 106 | |
Benjamin Peterson | 344ff4a | 2014-08-19 16:13:26 -0500 | [diff] [blame] | 107 | |
Éric Araujo | 01606de | 2011-03-26 03:22:55 +0100 | [diff] [blame] | 108 | There is no command-line option to control the optimization level used by the |
| 109 | :func:`compile` function, because the Python interpreter itself already |
| 110 | provides the option: :program:`python -O -m compileall`. |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 111 | |
Carl Meyer | b193fa9 | 2018-06-15 22:40:56 -0600 | [diff] [blame] | 112 | Similarly, the :func:`compile` function respects the :attr:`sys.pycache_prefix` |
| 113 | setting. The generated bytecode cache will only be useful if :func:`compile` is |
| 114 | run with the same :attr:`sys.pycache_prefix` (if any) that will be used at |
| 115 | runtime. |
| 116 | |
Éric Araujo | 713d303 | 2010-11-18 16:38:46 +0000 | [diff] [blame] | 117 | Public functions |
| 118 | ---------------- |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 119 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 120 | .. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 121 | |
| 122 | Recursively descend the directory tree named by *dir*, compiling all :file:`.py` |
Brett Cannon | 1e3c3e9 | 2015-12-27 13:17:04 -0800 | [diff] [blame] | 123 | files along the way. Return a true value if all the files compiled successfully, |
| 124 | and a false value otherwise. |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 125 | |
| 126 | The *maxlevels* parameter is used to limit the depth of the recursion; it |
| 127 | defaults to ``10``. |
| 128 | |
| 129 | If *ddir* is given, it is prepended to the path to each file being compiled |
| 130 | for use in compilation time tracebacks, and is also compiled in to the |
| 131 | byte-code file, where it will be used in tracebacks and other messages in |
| 132 | cases where the source file does not exist at the time the byte-code file is |
| 133 | executed. |
| 134 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 135 | If *force* is true, modules are re-compiled even if the timestamps are up to |
| 136 | date. |
| 137 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 138 | If *rx* is given, its search method is called on the complete path to each |
| 139 | file considered for compilation, and if it returns a true value, the file |
| 140 | is skipped. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 141 | |
Berker Peksag | 6554b86 | 2014-10-15 11:10:57 +0300 | [diff] [blame] | 142 | If *quiet* is ``False`` or ``0`` (the default), the filenames and other |
| 143 | information are printed to standard out. Set to ``1``, only errors are |
| 144 | printed. Set to ``2``, all output is suppressed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 145 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 146 | If *legacy* is true, byte-code files are written to their legacy locations |
| 147 | and names, which may overwrite byte-code files created by another version of |
| 148 | Python. The default is to write files to their :pep:`3147` locations and |
| 149 | names, which allows byte-code files from multiple versions of Python to |
| 150 | coexist. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 151 | |
Georg Brandl | 8334fd9 | 2010-12-04 10:26:46 +0000 | [diff] [blame] | 152 | *optimize* specifies the optimization level for the compiler. It is passed to |
| 153 | the built-in :func:`compile` function. |
Barry Warsaw | 28a691b | 2010-04-17 00:19:56 +0000 | [diff] [blame] | 154 | |
Brett Cannon | f1a8df0 | 2014-09-12 10:39:48 -0400 | [diff] [blame] | 155 | The argument *workers* specifies how many workers are used to |
| 156 | compile files in parallel. The default is to not use multiple workers. |
| 157 | If the platform can't use multiple workers and *workers* argument is given, |
Berker Peksag | d86ef05 | 2015-04-22 09:39:19 +0300 | [diff] [blame] | 158 | then sequential compilation will be used as a fallback. If *workers* is |
| 159 | lower than ``0``, a :exc:`ValueError` will be raised. |
Brett Cannon | f1a8df0 | 2014-09-12 10:39:48 -0400 | [diff] [blame] | 160 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 161 | *invalidation_mode* should be a member of the |
| 162 | :class:`py_compile.PycInvalidationMode` enum and controls how the generated |
| 163 | pycs are invalidated at runtime. |
| 164 | |
Georg Brandl | 8334fd9 | 2010-12-04 10:26:46 +0000 | [diff] [blame] | 165 | .. versionchanged:: 3.2 |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 166 | Added the *legacy* and *optimize* parameter. |
| 167 | |
Brett Cannon | f1a8df0 | 2014-09-12 10:39:48 -0400 | [diff] [blame] | 168 | .. versionchanged:: 3.5 |
| 169 | Added the *workers* parameter. |
| 170 | |
Berker Peksag | 6554b86 | 2014-10-15 11:10:57 +0300 | [diff] [blame] | 171 | .. versionchanged:: 3.5 |
| 172 | *quiet* parameter was changed to a multilevel value. |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 173 | |
Brett Cannon | f299abd | 2015-04-13 14:21:02 -0400 | [diff] [blame] | 174 | .. versionchanged:: 3.5 |
| 175 | The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files |
| 176 | no matter what the value of *optimize* is. |
| 177 | |
Berker Peksag | 812a2b6 | 2016-10-01 00:54:18 +0300 | [diff] [blame] | 178 | .. versionchanged:: 3.6 |
| 179 | Accepts a :term:`path-like object`. |
| 180 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 181 | .. versionchanged:: 3.7 |
| 182 | The *invalidation_mode* parameter was added. |
| 183 | |
| 184 | .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP) |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 185 | |
Brett Cannon | 1e3c3e9 | 2015-12-27 13:17:04 -0800 | [diff] [blame] | 186 | Compile the file with path *fullname*. Return a true value if the file |
| 187 | compiled successfully, and a false value otherwise. |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 188 | |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 189 | If *ddir* is given, it is prepended to the path to the file being compiled |
| 190 | for use in compilation time tracebacks, and is also compiled in to the |
| 191 | byte-code file, where it will be used in tracebacks and other messages in |
| 192 | cases where the source file does not exist at the time the byte-code file is |
| 193 | executed. |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 194 | |
R. David Murray | 8b24aac | 2011-02-11 22:37:16 +0000 | [diff] [blame] | 195 | If *rx* is given, its search method is passed the full path name to the |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 196 | file being compiled, and if it returns a true value, the file is not |
| 197 | compiled and ``True`` is returned. |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 198 | |
Berker Peksag | 6554b86 | 2014-10-15 11:10:57 +0300 | [diff] [blame] | 199 | If *quiet* is ``False`` or ``0`` (the default), the filenames and other |
| 200 | information are printed to standard out. Set to ``1``, only errors are |
| 201 | printed. Set to ``2``, all output is suppressed. |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 202 | |
| 203 | If *legacy* is true, byte-code files are written to their legacy locations |
| 204 | and names, which may overwrite byte-code files created by another version of |
| 205 | Python. The default is to write files to their :pep:`3147` locations and |
| 206 | names, which allows byte-code files from multiple versions of Python to |
| 207 | coexist. |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 208 | |
| 209 | *optimize* specifies the optimization level for the compiler. It is passed to |
| 210 | the built-in :func:`compile` function. |
| 211 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 212 | *invalidation_mode* should be a member of the |
| 213 | :class:`py_compile.PycInvalidationMode` enum and controls how the generated |
| 214 | pycs are invalidated at runtime. |
| 215 | |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 216 | .. versionadded:: 3.2 |
Georg Brandl | 8334fd9 | 2010-12-04 10:26:46 +0000 | [diff] [blame] | 217 | |
Berker Peksag | 6554b86 | 2014-10-15 11:10:57 +0300 | [diff] [blame] | 218 | .. versionchanged:: 3.5 |
| 219 | *quiet* parameter was changed to a multilevel value. |
Georg Brandl | 8334fd9 | 2010-12-04 10:26:46 +0000 | [diff] [blame] | 220 | |
Brett Cannon | f299abd | 2015-04-13 14:21:02 -0400 | [diff] [blame] | 221 | .. versionchanged:: 3.5 |
| 222 | The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files |
| 223 | no matter what the value of *optimize* is. |
| 224 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 225 | .. versionchanged:: 3.7 |
| 226 | The *invalidation_mode* parameter was added. |
| 227 | |
| 228 | .. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 229 | |
Brett Cannon | 1e3c3e9 | 2015-12-27 13:17:04 -0800 | [diff] [blame] | 230 | Byte-compile all the :file:`.py` files found along ``sys.path``. Return a |
| 231 | true value if all the files compiled successfully, and a false value otherwise. |
| 232 | |
| 233 | If *skip_curdir* is true (the default), the current directory is not included |
R. David Murray | 94f58c3 | 2010-12-17 16:29:07 +0000 | [diff] [blame] | 234 | in the search. All other parameters are passed to the :func:`compile_dir` |
| 235 | function. Note that unlike the other compile functions, ``maxlevels`` |
| 236 | defaults to ``0``. |
Georg Brandl | 8334fd9 | 2010-12-04 10:26:46 +0000 | [diff] [blame] | 237 | |
| 238 | .. versionchanged:: 3.2 |
Éric Araujo | 930df31 | 2010-12-16 06:28:48 +0000 | [diff] [blame] | 239 | Added the *legacy* and *optimize* parameter. |
Georg Brandl | 8334fd9 | 2010-12-04 10:26:46 +0000 | [diff] [blame] | 240 | |
Berker Peksag | 6554b86 | 2014-10-15 11:10:57 +0300 | [diff] [blame] | 241 | .. versionchanged:: 3.5 |
| 242 | *quiet* parameter was changed to a multilevel value. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 243 | |
Brett Cannon | f299abd | 2015-04-13 14:21:02 -0400 | [diff] [blame] | 244 | .. versionchanged:: 3.5 |
| 245 | The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files |
| 246 | no matter what the value of *optimize* is. |
| 247 | |
Benjamin Peterson | 42aa93b | 2017-12-09 10:26:52 -0800 | [diff] [blame] | 248 | .. versionchanged:: 3.7 |
| 249 | The *invalidation_mode* parameter was added. |
| 250 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 251 | To force a recompile of all the :file:`.py` files in the :file:`Lib/` |
| 252 | subdirectory and all its subdirectories:: |
| 253 | |
| 254 | import compileall |
| 255 | |
| 256 | compileall.compile_dir('Lib/', force=True) |
| 257 | |
| 258 | # Perform same compilation, excluding files in .svn directories. |
| 259 | import re |
Georg Brandl | 1aca953 | 2013-04-14 12:02:43 +0200 | [diff] [blame] | 260 | compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 261 | |
Berker Peksag | 812a2b6 | 2016-10-01 00:54:18 +0300 | [diff] [blame] | 262 | # pathlib.Path objects can also be used. |
| 263 | import pathlib |
| 264 | compileall.compile_dir(pathlib.Path('Lib/'), force=True) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 265 | |
| 266 | .. seealso:: |
| 267 | |
| 268 | Module :mod:`py_compile` |
| 269 | Byte-compile a single source file. |