Antoine Pitrou | 64a467d | 2010-12-12 20:34:49 +0000 | [diff] [blame] | 1 | :mod:`concurrent.futures` --- Launching parallel tasks |
| 2 | ====================================================== |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 3 | |
| 4 | .. module:: concurrent.futures |
| 5 | :synopsis: Execute computations concurrently using threads or processes. |
| 6 | |
Éric Araujo | 19f9b71 | 2011-08-19 00:49:18 +0200 | [diff] [blame] | 7 | .. versionadded:: 3.2 |
| 8 | |
Raymond Hettinger | a199368 | 2011-01-27 01:20:32 +0000 | [diff] [blame] | 9 | **Source code:** :source:`Lib/concurrent/futures/thread.py` |
| 10 | and :source:`Lib/concurrent/futures/process.py` |
| 11 | |
Raymond Hettinger | a199368 | 2011-01-27 01:20:32 +0000 | [diff] [blame] | 12 | -------------- |
| 13 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 14 | The :mod:`concurrent.futures` module provides a high-level interface for |
| 15 | asynchronously executing callables. |
| 16 | |
Ezio Melotti | e130a52 | 2011-10-19 10:58:56 +0300 | [diff] [blame] | 17 | The asynchronous execution can be performed with threads, using |
Georg Brandl | fb1720b | 2010-12-09 18:08:43 +0000 | [diff] [blame] | 18 | :class:`ThreadPoolExecutor`, or separate processes, using |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 19 | :class:`ProcessPoolExecutor`. Both implement the same interface, which is |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 20 | defined by the abstract :class:`Executor` class. |
| 21 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 22 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 23 | Executor Objects |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 24 | ---------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 25 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 26 | .. class:: Executor |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 27 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 28 | An abstract class that provides methods to execute calls asynchronously. It |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 29 | should not be used directly, but through its concrete subclasses. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 30 | |
| 31 | .. method:: submit(fn, *args, **kwargs) |
| 32 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 33 | Schedules the callable, *fn*, to be executed as ``fn(*args **kwargs)`` |
| 34 | and returns a :class:`Future` object representing the execution of the |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 35 | callable. :: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 36 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 37 | with ThreadPoolExecutor(max_workers=1) as executor: |
| 38 | future = executor.submit(pow, 323, 1235) |
| 39 | print(future.result()) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 40 | |
Antoine Pitrou | 4aae276 | 2014-10-04 20:20:10 +0200 | [diff] [blame] | 41 | .. method:: map(func, *iterables, timeout=None, chunksize=1) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 42 | |
Antoine Pitrou | a7a751d | 2017-12-20 19:06:20 +0100 | [diff] [blame] | 43 | Similar to :func:`map(func, *iterables) <map>` except: |
| 44 | |
| 45 | * the *iterables* are collected immediately rather than lazily; |
| 46 | |
| 47 | * *func* is executed asynchronously and several calls to |
| 48 | *func* may be made concurrently. |
| 49 | |
| 50 | The returned iterator raises a :exc:`concurrent.futures.TimeoutError` |
| 51 | if :meth:`~iterator.__next__` is called and the result isn't available |
Ezio Melotti | 7fa8222 | 2012-10-12 13:42:08 +0300 | [diff] [blame] | 52 | after *timeout* seconds from the original call to :meth:`Executor.map`. |
| 53 | *timeout* can be an int or a float. If *timeout* is not specified or |
Antoine Pitrou | a7a751d | 2017-12-20 19:06:20 +0100 | [diff] [blame] | 54 | ``None``, there is no limit to the wait time. |
| 55 | |
| 56 | If a *func* call raises an exception, then that exception will be |
| 57 | raised when its value is retrieved from the iterator. |
| 58 | |
| 59 | When using :class:`ProcessPoolExecutor`, this method chops *iterables* |
| 60 | into a number of chunks which it submits to the pool as separate |
| 61 | tasks. The (approximate) size of these chunks can be specified by |
| 62 | setting *chunksize* to a positive integer. For very long iterables, |
| 63 | using a large value for *chunksize* can significantly improve |
| 64 | performance compared to the default size of 1. With |
| 65 | :class:`ThreadPoolExecutor`, *chunksize* has no effect. |
Antoine Pitrou | 4aae276 | 2014-10-04 20:20:10 +0200 | [diff] [blame] | 66 | |
| 67 | .. versionchanged:: 3.5 |
| 68 | Added the *chunksize* argument. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 69 | |
| 70 | .. method:: shutdown(wait=True) |
| 71 | |
| 72 | Signal the executor that it should free any resources that it is using |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 73 | when the currently pending futures are done executing. Calls to |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 74 | :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will |
| 75 | raise :exc:`RuntimeError`. |
| 76 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 77 | If *wait* is ``True`` then this method will not return until all the |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 78 | pending futures are done executing and the resources associated with the |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 79 | executor have been freed. If *wait* is ``False`` then this method will |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 80 | return immediately and the resources associated with the executor will be |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 81 | freed when all pending futures are done executing. Regardless of the |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 82 | value of *wait*, the entire Python program will not exit until all |
| 83 | pending futures are done executing. |
| 84 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 85 | You can avoid having to call this method explicitly if you use the |
| 86 | :keyword:`with` statement, which will shutdown the :class:`Executor` |
| 87 | (waiting as if :meth:`Executor.shutdown` were called with *wait* set to |
| 88 | ``True``):: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 89 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 90 | import shutil |
| 91 | with ThreadPoolExecutor(max_workers=4) as e: |
| 92 | e.submit(shutil.copy, 'src1.txt', 'dest1.txt') |
| 93 | e.submit(shutil.copy, 'src2.txt', 'dest2.txt') |
| 94 | e.submit(shutil.copy, 'src3.txt', 'dest3.txt') |
Berker Peksag | 0b0c3b6 | 2015-09-15 19:59:03 +0300 | [diff] [blame] | 95 | e.submit(shutil.copy, 'src4.txt', 'dest4.txt') |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 96 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 97 | |
| 98 | ThreadPoolExecutor |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 99 | ------------------ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 100 | |
Martin Panter | 7462b649 | 2015-11-02 03:37:02 +0000 | [diff] [blame] | 101 | :class:`ThreadPoolExecutor` is an :class:`Executor` subclass that uses a pool of |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 102 | threads to execute calls asynchronously. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 103 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 104 | Deadlocks can occur when the callable associated with a :class:`Future` waits on |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 105 | the results of another :class:`Future`. For example:: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 106 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 107 | import time |
| 108 | def wait_on_b(): |
| 109 | time.sleep(5) |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 110 | print(b.result()) # b will never complete because it is waiting on a. |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 111 | return 5 |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 112 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 113 | def wait_on_a(): |
| 114 | time.sleep(5) |
Serhiy Storchaka | dba9039 | 2016-05-10 12:01:23 +0300 | [diff] [blame] | 115 | print(a.result()) # a will never complete because it is waiting on b. |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 116 | return 6 |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 117 | |
| 118 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 119 | executor = ThreadPoolExecutor(max_workers=2) |
| 120 | a = executor.submit(wait_on_b) |
| 121 | b = executor.submit(wait_on_a) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 122 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 123 | And:: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 124 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 125 | def wait_on_future(): |
| 126 | f = executor.submit(pow, 5, 2) |
| 127 | # This will never complete because there is only one worker thread and |
| 128 | # it is executing this function. |
| 129 | print(f.result()) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 130 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 131 | executor = ThreadPoolExecutor(max_workers=1) |
| 132 | executor.submit(wait_on_future) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 133 | |
| 134 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 135 | .. class:: ThreadPoolExecutor(max_workers=None, thread_name_prefix='', initializer=None, initargs=()) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 136 | |
| 137 | An :class:`Executor` subclass that uses a pool of at most *max_workers* |
| 138 | threads to execute calls asynchronously. |
| 139 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 140 | *initializer* is an optional callable that is called at the start of |
| 141 | each worker thread; *initargs* is a tuple of arguments passed to the |
| 142 | initializer. Should *initializer* raise an exception, all currently |
| 143 | pending jobs will raise a :exc:`~concurrent.futures.thread.BrokenThreadPool`, |
| 144 | as well any attempt to submit more jobs to the pool. |
| 145 | |
Guido van Rossum | cfd4661 | 2014-09-02 10:39:18 -0700 | [diff] [blame] | 146 | .. versionchanged:: 3.5 |
| 147 | If *max_workers* is ``None`` or |
| 148 | not given, it will default to the number of processors on the machine, |
| 149 | multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often |
| 150 | used to overlap I/O instead of CPU work and the number of workers |
| 151 | should be higher than the number of workers |
| 152 | for :class:`ProcessPoolExecutor`. |
| 153 | |
Gregory P. Smith | 50abe87 | 2016-08-07 10:19:20 -0700 | [diff] [blame] | 154 | .. versionadded:: 3.6 |
| 155 | The *thread_name_prefix* argument was added to allow users to |
| 156 | control the threading.Thread names for worker threads created by |
| 157 | the pool for easier debugging. |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 158 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 159 | .. versionchanged:: 3.7 |
| 160 | Added the *initializer* and *initargs* arguments. |
| 161 | |
| 162 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 163 | .. _threadpoolexecutor-example: |
| 164 | |
| 165 | ThreadPoolExecutor Example |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 166 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 167 | :: |
| 168 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 169 | import concurrent.futures |
| 170 | import urllib.request |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 171 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 172 | URLS = ['http://www.foxnews.com/', |
| 173 | 'http://www.cnn.com/', |
| 174 | 'http://europe.wsj.com/', |
| 175 | 'http://www.bbc.co.uk/', |
| 176 | 'http://some-made-up-domain.com/'] |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 177 | |
Martin Panter | fe289c0 | 2016-05-28 02:20:39 +0000 | [diff] [blame] | 178 | # Retrieve a single page and report the URL and contents |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 179 | def load_url(url, timeout): |
Berker Peksag | 9575e18 | 2015-04-12 13:52:49 +0300 | [diff] [blame] | 180 | with urllib.request.urlopen(url, timeout=timeout) as conn: |
| 181 | return conn.read() |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 182 | |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 183 | # We can use a with statement to ensure threads are cleaned up promptly |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 184 | with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 185 | # Start the load operations and mark each future with its URL |
Georg Brandl | 0a1bc11 | 2013-03-23 15:59:46 +0100 | [diff] [blame] | 186 | future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} |
Nick Coghlan | d6d5cf3 | 2012-10-16 23:14:03 +1000 | [diff] [blame] | 187 | for future in concurrent.futures.as_completed(future_to_url): |
Nick Coghlan | 40c6773 | 2012-10-20 20:13:21 +1000 | [diff] [blame] | 188 | url = future_to_url[future] |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 189 | try: |
| 190 | data = future.result() |
| 191 | except Exception as exc: |
| 192 | print('%r generated an exception: %s' % (url, exc)) |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 193 | else: |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 194 | print('%r page is %d bytes' % (url, len(data))) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 195 | |
| 196 | |
| 197 | ProcessPoolExecutor |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 198 | ------------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 199 | |
| 200 | The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that |
| 201 | uses a pool of processes to execute calls asynchronously. |
| 202 | :class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which |
| 203 | allows it to side-step the :term:`Global Interpreter Lock` but also means that |
| 204 | only picklable objects can be executed and returned. |
| 205 | |
bquinlan | 7749cb5 | 2013-10-26 04:49:55 +1100 | [diff] [blame] | 206 | The ``__main__`` module must be importable by worker subprocesses. This means |
| 207 | that :class:`ProcessPoolExecutor` will not work in the interactive interpreter. |
| 208 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 209 | Calling :class:`Executor` or :class:`Future` methods from a callable submitted |
| 210 | to a :class:`ProcessPoolExecutor` will result in deadlock. |
| 211 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 212 | .. class:: ProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=()) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 213 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 214 | An :class:`Executor` subclass that executes calls asynchronously using a pool |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 215 | of at most *max_workers* processes. If *max_workers* is ``None`` or not |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 216 | given, it will default to the number of processors on the machine. |
Brian Quinlan | 20efceb | 2014-05-17 13:51:10 -0700 | [diff] [blame] | 217 | If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` |
| 218 | will be raised. |
Thomas Moreau | e8c368d | 2017-10-03 11:53:17 +0200 | [diff] [blame] | 219 | *mp_context* can be a multiprocessing context or None. It will be used to |
| 220 | launch the workers. If *mp_context* is ``None`` or not given, the default |
| 221 | multiprocessing context is used. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 222 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 223 | *initializer* is an optional callable that is called at the start of |
| 224 | each worker process; *initargs* is a tuple of arguments passed to the |
| 225 | initializer. Should *initializer* raise an exception, all currently |
| 226 | pending jobs will raise a :exc:`~concurrent.futures.thread.BrokenThreadPool`, |
| 227 | as well any attempt to submit more jobs to the pool. |
| 228 | |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 229 | .. versionchanged:: 3.3 |
| 230 | When one of the worker processes terminates abruptly, a |
| 231 | :exc:`BrokenProcessPool` error is now raised. Previously, behaviour |
| 232 | was undefined but operations on the executor or its futures would often |
| 233 | freeze or deadlock. |
| 234 | |
Thomas Moreau | e8c368d | 2017-10-03 11:53:17 +0200 | [diff] [blame] | 235 | .. versionchanged:: 3.7 |
| 236 | The *mp_context* argument was added to allow users to control the |
| 237 | start_method for worker processes created by the pool. |
| 238 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 239 | Added the *initializer* and *initargs* arguments. |
| 240 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 241 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 242 | .. _processpoolexecutor-example: |
| 243 | |
| 244 | ProcessPoolExecutor Example |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 245 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 246 | :: |
| 247 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 248 | import concurrent.futures |
| 249 | import math |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 250 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 251 | PRIMES = [ |
| 252 | 112272535095293, |
| 253 | 112582705942171, |
| 254 | 112272535095293, |
| 255 | 115280095190773, |
| 256 | 115797848077099, |
| 257 | 1099726899285419] |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 258 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 259 | def is_prime(n): |
| 260 | if n % 2 == 0: |
| 261 | return False |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 262 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 263 | sqrt_n = int(math.floor(math.sqrt(n))) |
| 264 | for i in range(3, sqrt_n + 1, 2): |
| 265 | if n % i == 0: |
| 266 | return False |
| 267 | return True |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 268 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 269 | def main(): |
| 270 | with concurrent.futures.ProcessPoolExecutor() as executor: |
| 271 | for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): |
| 272 | print('%d is prime: %s' % (number, prime)) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 273 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 274 | if __name__ == '__main__': |
| 275 | main() |
| 276 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 277 | |
| 278 | Future Objects |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 279 | -------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 280 | |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 281 | The :class:`Future` class encapsulates the asynchronous execution of a callable. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 282 | :class:`Future` instances are created by :meth:`Executor.submit`. |
| 283 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 284 | .. class:: Future |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 285 | |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 286 | Encapsulates the asynchronous execution of a callable. :class:`Future` |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 287 | instances are created by :meth:`Executor.submit` and should not be created |
| 288 | directly except for testing. |
| 289 | |
| 290 | .. method:: cancel() |
| 291 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 292 | Attempt to cancel the call. If the call is currently being executed and |
Eric Smith | 7b5011b | 2011-02-01 21:31:22 +0000 | [diff] [blame] | 293 | cannot be cancelled then the method will return ``False``, otherwise the |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 294 | call will be cancelled and the method will return ``True``. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 295 | |
| 296 | .. method:: cancelled() |
| 297 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 298 | Return ``True`` if the call was successfully cancelled. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 299 | |
| 300 | .. method:: running() |
| 301 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 302 | Return ``True`` if the call is currently being executed and cannot be |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 303 | cancelled. |
| 304 | |
| 305 | .. method:: done() |
| 306 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 307 | Return ``True`` if the call was successfully cancelled or finished |
| 308 | running. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 309 | |
| 310 | .. method:: result(timeout=None) |
| 311 | |
| 312 | Return the value returned by the call. If the call hasn't yet completed |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 313 | then this method will wait up to *timeout* seconds. If the call hasn't |
Senthil Kumaran | 9e9f850 | 2016-01-18 18:45:00 -0800 | [diff] [blame] | 314 | completed in *timeout* seconds, then a |
| 315 | :exc:`concurrent.futures.TimeoutError` will be raised. *timeout* can be |
| 316 | an int or float. If *timeout* is not specified or ``None``, there is no |
| 317 | limit to the wait time. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 318 | |
Senthil Kumaran | 9e9f850 | 2016-01-18 18:45:00 -0800 | [diff] [blame] | 319 | If the future is cancelled before completing then :exc:`.CancelledError` |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 320 | will be raised. |
| 321 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 322 | If the call raised, this method will raise the same exception. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 323 | |
| 324 | .. method:: exception(timeout=None) |
| 325 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 326 | Return the exception raised by the call. If the call hasn't yet |
| 327 | completed then this method will wait up to *timeout* seconds. If the |
Senthil Kumaran | 9e9f850 | 2016-01-18 18:45:00 -0800 | [diff] [blame] | 328 | call hasn't completed in *timeout* seconds, then a |
| 329 | :exc:`concurrent.futures.TimeoutError` will be raised. *timeout* can be |
| 330 | an int or float. If *timeout* is not specified or ``None``, there is no |
| 331 | limit to the wait time. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 332 | |
Senthil Kumaran | 9e9f850 | 2016-01-18 18:45:00 -0800 | [diff] [blame] | 333 | If the future is cancelled before completing then :exc:`.CancelledError` |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 334 | will be raised. |
| 335 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 336 | If the call completed without raising, ``None`` is returned. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 337 | |
| 338 | .. method:: add_done_callback(fn) |
| 339 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 340 | Attaches the callable *fn* to the future. *fn* will be called, with the |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 341 | future as its only argument, when the future is cancelled or finishes |
| 342 | running. |
| 343 | |
| 344 | Added callables are called in the order that they were added and are |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 345 | always called in a thread belonging to the process that added them. If |
Martin Panter | 7462b649 | 2015-11-02 03:37:02 +0000 | [diff] [blame] | 346 | the callable raises an :exc:`Exception` subclass, it will be logged and |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 347 | ignored. If the callable raises a :exc:`BaseException` subclass, the |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 348 | behavior is undefined. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 349 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 350 | If the future has already completed or been cancelled, *fn* will be |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 351 | called immediately. |
| 352 | |
| 353 | The following :class:`Future` methods are meant for use in unit tests and |
| 354 | :class:`Executor` implementations. |
| 355 | |
| 356 | .. method:: set_running_or_notify_cancel() |
| 357 | |
| 358 | This method should only be called by :class:`Executor` implementations |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 359 | before executing the work associated with the :class:`Future` and by unit |
| 360 | tests. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 361 | |
Senthil Kumaran | 916bd38 | 2010-10-15 12:55:19 +0000 | [diff] [blame] | 362 | If the method returns ``False`` then the :class:`Future` was cancelled, |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 363 | i.e. :meth:`Future.cancel` was called and returned `True`. Any threads |
| 364 | waiting on the :class:`Future` completing (i.e. through |
| 365 | :func:`as_completed` or :func:`wait`) will be woken up. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 366 | |
Senthil Kumaran | 916bd38 | 2010-10-15 12:55:19 +0000 | [diff] [blame] | 367 | If the method returns ``True`` then the :class:`Future` was not cancelled |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 368 | and has been put in the running state, i.e. calls to |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 369 | :meth:`Future.running` will return `True`. |
| 370 | |
| 371 | This method can only be called once and cannot be called after |
| 372 | :meth:`Future.set_result` or :meth:`Future.set_exception` have been |
| 373 | called. |
| 374 | |
| 375 | .. method:: set_result(result) |
| 376 | |
| 377 | Sets the result of the work associated with the :class:`Future` to |
| 378 | *result*. |
| 379 | |
| 380 | This method should only be used by :class:`Executor` implementations and |
| 381 | unit tests. |
| 382 | |
jhaydaman | 0a28c0d | 2018-05-30 02:15:06 -0500 | [diff] [blame] | 383 | .. versionchanged:: 3.8 |
| 384 | This method raises |
| 385 | :exc:`concurrent.futures.InvalidStateError` if the :class:`Future` is |
| 386 | already done. |
| 387 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 388 | .. method:: set_exception(exception) |
| 389 | |
| 390 | Sets the result of the work associated with the :class:`Future` to the |
| 391 | :class:`Exception` *exception*. |
| 392 | |
| 393 | This method should only be used by :class:`Executor` implementations and |
| 394 | unit tests. |
| 395 | |
jhaydaman | 0a28c0d | 2018-05-30 02:15:06 -0500 | [diff] [blame] | 396 | .. versionchanged:: 3.8 |
| 397 | This method raises |
| 398 | :exc:`concurrent.futures.InvalidStateError` if the :class:`Future` is |
| 399 | already done. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 400 | |
| 401 | Module Functions |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 402 | ---------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 403 | |
| 404 | .. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) |
| 405 | |
| 406 | Wait for the :class:`Future` instances (possibly created by different |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 407 | :class:`Executor` instances) given by *fs* to complete. Returns a named |
| 408 | 2-tuple of sets. The first set, named ``done``, contains the futures that |
| 409 | completed (finished or were cancelled) before the wait completed. The second |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 410 | set, named ``not_done``, contains uncompleted futures. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 411 | |
| 412 | *timeout* can be used to control the maximum number of seconds to wait before |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 413 | returning. *timeout* can be an int or float. If *timeout* is not specified |
| 414 | or ``None``, there is no limit to the wait time. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 415 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 416 | *return_when* indicates when this function should return. It must be one of |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 417 | the following constants: |
| 418 | |
Georg Brandl | 44ea77b | 2013-03-28 13:28:44 +0100 | [diff] [blame] | 419 | .. tabularcolumns:: |l|L| |
| 420 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 421 | +-----------------------------+----------------------------------------+ |
| 422 | | Constant | Description | |
| 423 | +=============================+========================================+ |
| 424 | | :const:`FIRST_COMPLETED` | The function will return when any | |
| 425 | | | future finishes or is cancelled. | |
| 426 | +-----------------------------+----------------------------------------+ |
| 427 | | :const:`FIRST_EXCEPTION` | The function will return when any | |
| 428 | | | future finishes by raising an | |
| 429 | | | exception. If no future raises an | |
| 430 | | | exception then it is equivalent to | |
| 431 | | | :const:`ALL_COMPLETED`. | |
| 432 | +-----------------------------+----------------------------------------+ |
| 433 | | :const:`ALL_COMPLETED` | The function will return when all | |
| 434 | | | futures finish or are cancelled. | |
| 435 | +-----------------------------+----------------------------------------+ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 436 | |
| 437 | .. function:: as_completed(fs, timeout=None) |
| 438 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 439 | Returns an iterator over the :class:`Future` instances (possibly created by |
| 440 | different :class:`Executor` instances) given by *fs* that yields futures as |
Guido van Rossum | e6994ff | 2014-01-26 09:57:51 -0800 | [diff] [blame] | 441 | they complete (finished or were cancelled). Any futures given by *fs* that |
Senthil Kumaran | 9e9f850 | 2016-01-18 18:45:00 -0800 | [diff] [blame] | 442 | are duplicated will be returned once. Any futures that completed before |
| 443 | :func:`as_completed` is called will be yielded first. The returned iterator |
| 444 | raises a :exc:`concurrent.futures.TimeoutError` if :meth:`~iterator.__next__` |
| 445 | is called and the result isn't available after *timeout* seconds from the |
| 446 | original call to :func:`as_completed`. *timeout* can be an int or float. If |
| 447 | *timeout* is not specified or ``None``, there is no limit to the wait time. |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 448 | |
| 449 | |
| 450 | .. seealso:: |
| 451 | |
| 452 | :pep:`3148` -- futures - execute computations asynchronously |
| 453 | The proposal which described this feature for inclusion in the Python |
| 454 | standard library. |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 455 | |
| 456 | |
| 457 | Exception classes |
| 458 | ----------------- |
| 459 | |
Senthil Kumaran | 9e9f850 | 2016-01-18 18:45:00 -0800 | [diff] [blame] | 460 | .. currentmodule:: concurrent.futures |
| 461 | |
| 462 | .. exception:: CancelledError |
| 463 | |
| 464 | Raised when a future is cancelled. |
| 465 | |
| 466 | .. exception:: TimeoutError |
| 467 | |
| 468 | Raised when a future operation exceeds the given timeout. |
| 469 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 470 | .. exception:: BrokenExecutor |
| 471 | |
| 472 | Derived from :exc:`RuntimeError`, this exception class is raised |
| 473 | when an executor is broken for some reason, and cannot be used |
| 474 | to submit or execute new tasks. |
| 475 | |
| 476 | .. versionadded:: 3.7 |
| 477 | |
jhaydaman | 0a28c0d | 2018-05-30 02:15:06 -0500 | [diff] [blame] | 478 | .. exception:: InvalidStateError |
| 479 | |
| 480 | Raised when an operation is performed on a future that is not allowed |
| 481 | in the current state. |
| 482 | |
| 483 | .. versionadded:: 3.8 |
| 484 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 485 | .. currentmodule:: concurrent.futures.thread |
| 486 | |
| 487 | .. exception:: BrokenThreadPool |
| 488 | |
| 489 | Derived from :exc:`~concurrent.futures.BrokenExecutor`, this exception |
| 490 | class is raised when one of the workers of a :class:`ThreadPoolExecutor` |
| 491 | has failed initializing. |
| 492 | |
| 493 | .. versionadded:: 3.7 |
| 494 | |
Georg Brandl | ce64ced | 2014-10-28 22:58:24 +0100 | [diff] [blame] | 495 | .. currentmodule:: concurrent.futures.process |
| 496 | |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 497 | .. exception:: BrokenProcessPool |
| 498 | |
Antoine Pitrou | 63ff413 | 2017-11-04 11:05:49 +0100 | [diff] [blame] | 499 | Derived from :exc:`~concurrent.futures.BrokenExecutor` (formerly |
| 500 | :exc:`RuntimeError`), this exception class is raised when one of the |
| 501 | workers of a :class:`ProcessPoolExecutor` has terminated in a non-clean |
| 502 | fashion (for example, if it was killed from the outside). |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 503 | |
| 504 | .. versionadded:: 3.3 |