Antoine Pitrou | 64a467d | 2010-12-12 20:34:49 +0000 | [diff] [blame] | 1 | :mod:`concurrent.futures` --- Launching parallel tasks |
| 2 | ====================================================== |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 3 | |
| 4 | .. module:: concurrent.futures |
| 5 | :synopsis: Execute computations concurrently using threads or processes. |
| 6 | |
Éric Araujo | 19f9b71 | 2011-08-19 00:49:18 +0200 | [diff] [blame] | 7 | .. versionadded:: 3.2 |
| 8 | |
Raymond Hettinger | a199368 | 2011-01-27 01:20:32 +0000 | [diff] [blame] | 9 | **Source code:** :source:`Lib/concurrent/futures/thread.py` |
| 10 | and :source:`Lib/concurrent/futures/process.py` |
| 11 | |
Raymond Hettinger | a199368 | 2011-01-27 01:20:32 +0000 | [diff] [blame] | 12 | -------------- |
| 13 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 14 | The :mod:`concurrent.futures` module provides a high-level interface for |
| 15 | asynchronously executing callables. |
| 16 | |
Ezio Melotti | e130a52 | 2011-10-19 10:58:56 +0300 | [diff] [blame] | 17 | The asynchronous execution can be performed with threads, using |
Georg Brandl | fb1720b | 2010-12-09 18:08:43 +0000 | [diff] [blame] | 18 | :class:`ThreadPoolExecutor`, or separate processes, using |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 19 | :class:`ProcessPoolExecutor`. Both implement the same interface, which is |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 20 | defined by the abstract :class:`Executor` class. |
| 21 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 22 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 23 | Executor Objects |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 24 | ---------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 25 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 26 | .. class:: Executor |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 27 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 28 | An abstract class that provides methods to execute calls asynchronously. It |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 29 | should not be used directly, but through its concrete subclasses. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 30 | |
| 31 | .. method:: submit(fn, *args, **kwargs) |
| 32 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 33 | Schedules the callable, *fn*, to be executed as ``fn(*args **kwargs)`` |
| 34 | and returns a :class:`Future` object representing the execution of the |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 35 | callable. :: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 36 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 37 | with ThreadPoolExecutor(max_workers=1) as executor: |
| 38 | future = executor.submit(pow, 323, 1235) |
| 39 | print(future.result()) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 40 | |
| 41 | .. method:: map(func, *iterables, timeout=None) |
| 42 | |
Serhiy Storchaka | 9e0ae53 | 2013-08-24 00:23:38 +0300 | [diff] [blame] | 43 | Equivalent to :func:`map(func, *iterables) <map>` except *func* is executed |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 44 | asynchronously and several calls to *func* may be made concurrently. The |
Ezio Melotti | 7fa8222 | 2012-10-12 13:42:08 +0300 | [diff] [blame] | 45 | returned iterator raises a :exc:`TimeoutError` if |
| 46 | :meth:`~iterator.__next__` is called and the result isn't available |
| 47 | after *timeout* seconds from the original call to :meth:`Executor.map`. |
| 48 | *timeout* can be an int or a float. If *timeout* is not specified or |
| 49 | ``None``, there is no limit to the wait time. If a call raises an |
| 50 | exception, then that exception will be raised when its value is |
| 51 | retrieved from the iterator. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 52 | |
| 53 | .. method:: shutdown(wait=True) |
| 54 | |
| 55 | Signal the executor that it should free any resources that it is using |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 56 | when the currently pending futures are done executing. Calls to |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 57 | :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will |
| 58 | raise :exc:`RuntimeError`. |
| 59 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 60 | If *wait* is ``True`` then this method will not return until all the |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 61 | pending futures are done executing and the resources associated with the |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 62 | executor have been freed. If *wait* is ``False`` then this method will |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 63 | return immediately and the resources associated with the executor will be |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 64 | freed when all pending futures are done executing. Regardless of the |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 65 | value of *wait*, the entire Python program will not exit until all |
| 66 | pending futures are done executing. |
| 67 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 68 | You can avoid having to call this method explicitly if you use the |
| 69 | :keyword:`with` statement, which will shutdown the :class:`Executor` |
| 70 | (waiting as if :meth:`Executor.shutdown` were called with *wait* set to |
| 71 | ``True``):: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 72 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 73 | import shutil |
| 74 | with ThreadPoolExecutor(max_workers=4) as e: |
| 75 | e.submit(shutil.copy, 'src1.txt', 'dest1.txt') |
| 76 | e.submit(shutil.copy, 'src2.txt', 'dest2.txt') |
| 77 | e.submit(shutil.copy, 'src3.txt', 'dest3.txt') |
Berker Peksag | 0b0c3b6 | 2015-09-15 19:59:03 +0300 | [diff] [blame] | 78 | e.submit(shutil.copy, 'src4.txt', 'dest4.txt') |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 79 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 80 | |
| 81 | ThreadPoolExecutor |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 82 | ------------------ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 83 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 84 | :class:`ThreadPoolExecutor` is a :class:`Executor` subclass that uses a pool of |
| 85 | threads to execute calls asynchronously. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 86 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 87 | Deadlocks can occur when the callable associated with a :class:`Future` waits on |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 88 | the results of another :class:`Future`. For example:: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 89 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 90 | import time |
| 91 | def wait_on_b(): |
| 92 | time.sleep(5) |
| 93 | print(b.result()) # b will never complete because it is waiting on a. |
| 94 | return 5 |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 95 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 96 | def wait_on_a(): |
| 97 | time.sleep(5) |
| 98 | print(a.result()) # a will never complete because it is waiting on b. |
| 99 | return 6 |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 100 | |
| 101 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 102 | executor = ThreadPoolExecutor(max_workers=2) |
| 103 | a = executor.submit(wait_on_b) |
| 104 | b = executor.submit(wait_on_a) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 105 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 106 | And:: |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 107 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 108 | def wait_on_future(): |
| 109 | f = executor.submit(pow, 5, 2) |
| 110 | # This will never complete because there is only one worker thread and |
| 111 | # it is executing this function. |
| 112 | print(f.result()) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 113 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 114 | executor = ThreadPoolExecutor(max_workers=1) |
| 115 | executor.submit(wait_on_future) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 116 | |
| 117 | |
| 118 | .. class:: ThreadPoolExecutor(max_workers) |
| 119 | |
| 120 | An :class:`Executor` subclass that uses a pool of at most *max_workers* |
| 121 | threads to execute calls asynchronously. |
| 122 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 123 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 124 | .. _threadpoolexecutor-example: |
| 125 | |
| 126 | ThreadPoolExecutor Example |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 127 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 128 | :: |
| 129 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 130 | import concurrent.futures |
| 131 | import urllib.request |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 132 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 133 | URLS = ['http://www.foxnews.com/', |
| 134 | 'http://www.cnn.com/', |
| 135 | 'http://europe.wsj.com/', |
| 136 | 'http://www.bbc.co.uk/', |
| 137 | 'http://some-made-up-domain.com/'] |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 138 | |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 139 | # Retrieve a single page and report the url and contents |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 140 | def load_url(url, timeout): |
Berker Peksag | 9575e18 | 2015-04-12 13:52:49 +0300 | [diff] [blame] | 141 | with urllib.request.urlopen(url, timeout=timeout) as conn: |
| 142 | return conn.read() |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 143 | |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 144 | # We can use a with statement to ensure threads are cleaned up promptly |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 145 | with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 146 | # Start the load operations and mark each future with its URL |
Georg Brandl | 0a1bc11 | 2013-03-23 15:59:46 +0100 | [diff] [blame] | 147 | future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} |
Nick Coghlan | d6d5cf3 | 2012-10-16 23:14:03 +1000 | [diff] [blame] | 148 | for future in concurrent.futures.as_completed(future_to_url): |
Nick Coghlan | 40c6773 | 2012-10-20 20:13:21 +1000 | [diff] [blame] | 149 | url = future_to_url[future] |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 150 | try: |
| 151 | data = future.result() |
| 152 | except Exception as exc: |
| 153 | print('%r generated an exception: %s' % (url, exc)) |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 154 | else: |
Nick Coghlan | f06ea25 | 2012-10-16 22:50:04 +1000 | [diff] [blame] | 155 | print('%r page is %d bytes' % (url, len(data))) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 156 | |
| 157 | |
| 158 | ProcessPoolExecutor |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 159 | ------------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 160 | |
| 161 | The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that |
| 162 | uses a pool of processes to execute calls asynchronously. |
| 163 | :class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which |
| 164 | allows it to side-step the :term:`Global Interpreter Lock` but also means that |
| 165 | only picklable objects can be executed and returned. |
| 166 | |
bquinlan | 7749cb5 | 2013-10-26 04:49:55 +1100 | [diff] [blame] | 167 | The ``__main__`` module must be importable by worker subprocesses. This means |
| 168 | that :class:`ProcessPoolExecutor` will not work in the interactive interpreter. |
| 169 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 170 | Calling :class:`Executor` or :class:`Future` methods from a callable submitted |
| 171 | to a :class:`ProcessPoolExecutor` will result in deadlock. |
| 172 | |
| 173 | .. class:: ProcessPoolExecutor(max_workers=None) |
| 174 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 175 | An :class:`Executor` subclass that executes calls asynchronously using a pool |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 176 | of at most *max_workers* processes. If *max_workers* is ``None`` or not |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 177 | given, it will default to the number of processors on the machine. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 178 | |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 179 | .. versionchanged:: 3.3 |
| 180 | When one of the worker processes terminates abruptly, a |
| 181 | :exc:`BrokenProcessPool` error is now raised. Previously, behaviour |
| 182 | was undefined but operations on the executor or its futures would often |
| 183 | freeze or deadlock. |
| 184 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 185 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 186 | .. _processpoolexecutor-example: |
| 187 | |
| 188 | ProcessPoolExecutor Example |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 189 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 190 | :: |
| 191 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 192 | import concurrent.futures |
| 193 | import math |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 194 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 195 | PRIMES = [ |
| 196 | 112272535095293, |
| 197 | 112582705942171, |
| 198 | 112272535095293, |
| 199 | 115280095190773, |
| 200 | 115797848077099, |
| 201 | 1099726899285419] |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 202 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 203 | def is_prime(n): |
| 204 | if n % 2 == 0: |
| 205 | return False |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 206 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 207 | sqrt_n = int(math.floor(math.sqrt(n))) |
| 208 | for i in range(3, sqrt_n + 1, 2): |
| 209 | if n % i == 0: |
| 210 | return False |
| 211 | return True |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 212 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 213 | def main(): |
| 214 | with concurrent.futures.ProcessPoolExecutor() as executor: |
| 215 | for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): |
| 216 | print('%d is prime: %s' % (number, prime)) |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 217 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 218 | if __name__ == '__main__': |
| 219 | main() |
| 220 | |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 221 | |
| 222 | Future Objects |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 223 | -------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 224 | |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 225 | The :class:`Future` class encapsulates the asynchronous execution of a callable. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 226 | :class:`Future` instances are created by :meth:`Executor.submit`. |
| 227 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 228 | .. class:: Future |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 229 | |
Georg Brandl | 6faee4e | 2010-09-21 14:48:28 +0000 | [diff] [blame] | 230 | Encapsulates the asynchronous execution of a callable. :class:`Future` |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 231 | instances are created by :meth:`Executor.submit` and should not be created |
| 232 | directly except for testing. |
| 233 | |
| 234 | .. method:: cancel() |
| 235 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 236 | Attempt to cancel the call. If the call is currently being executed and |
Eric Smith | 7b5011b | 2011-02-01 21:31:22 +0000 | [diff] [blame] | 237 | cannot be cancelled then the method will return ``False``, otherwise the |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 238 | call will be cancelled and the method will return ``True``. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 239 | |
| 240 | .. method:: cancelled() |
| 241 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 242 | Return ``True`` if the call was successfully cancelled. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 243 | |
| 244 | .. method:: running() |
| 245 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 246 | Return ``True`` if the call is currently being executed and cannot be |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 247 | cancelled. |
| 248 | |
| 249 | .. method:: done() |
| 250 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 251 | Return ``True`` if the call was successfully cancelled or finished |
| 252 | running. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 253 | |
| 254 | .. method:: result(timeout=None) |
| 255 | |
| 256 | Return the value returned by the call. If the call hasn't yet completed |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 257 | then this method will wait up to *timeout* seconds. If the call hasn't |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 258 | completed in *timeout* seconds, then a :exc:`TimeoutError` will be |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 259 | raised. *timeout* can be an int or float. If *timeout* is not specified |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 260 | or ``None``, there is no limit to the wait time. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 261 | |
| 262 | If the future is cancelled before completing then :exc:`CancelledError` |
| 263 | will be raised. |
| 264 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 265 | If the call raised, this method will raise the same exception. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 266 | |
| 267 | .. method:: exception(timeout=None) |
| 268 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 269 | Return the exception raised by the call. If the call hasn't yet |
| 270 | completed then this method will wait up to *timeout* seconds. If the |
| 271 | call hasn't completed in *timeout* seconds, then a :exc:`TimeoutError` |
| 272 | will be raised. *timeout* can be an int or float. If *timeout* is not |
| 273 | specified or ``None``, there is no limit to the wait time. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 274 | |
| 275 | If the future is cancelled before completing then :exc:`CancelledError` |
| 276 | will be raised. |
| 277 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 278 | If the call completed without raising, ``None`` is returned. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 279 | |
| 280 | .. method:: add_done_callback(fn) |
| 281 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 282 | Attaches the callable *fn* to the future. *fn* will be called, with the |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 283 | future as its only argument, when the future is cancelled or finishes |
| 284 | running. |
| 285 | |
| 286 | Added callables are called in the order that they were added and are |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 287 | always called in a thread belonging to the process that added them. If |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 288 | the callable raises a :exc:`Exception` subclass, it will be logged and |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 289 | ignored. If the callable raises a :exc:`BaseException` subclass, the |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 290 | behavior is undefined. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 291 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 292 | If the future has already completed or been cancelled, *fn* will be |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 293 | called immediately. |
| 294 | |
| 295 | The following :class:`Future` methods are meant for use in unit tests and |
| 296 | :class:`Executor` implementations. |
| 297 | |
| 298 | .. method:: set_running_or_notify_cancel() |
| 299 | |
| 300 | This method should only be called by :class:`Executor` implementations |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 301 | before executing the work associated with the :class:`Future` and by unit |
| 302 | tests. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 303 | |
Senthil Kumaran | 916bd38 | 2010-10-15 12:55:19 +0000 | [diff] [blame] | 304 | If the method returns ``False`` then the :class:`Future` was cancelled, |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 305 | i.e. :meth:`Future.cancel` was called and returned `True`. Any threads |
| 306 | waiting on the :class:`Future` completing (i.e. through |
| 307 | :func:`as_completed` or :func:`wait`) will be woken up. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 308 | |
Senthil Kumaran | 916bd38 | 2010-10-15 12:55:19 +0000 | [diff] [blame] | 309 | If the method returns ``True`` then the :class:`Future` was not cancelled |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 310 | and has been put in the running state, i.e. calls to |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 311 | :meth:`Future.running` will return `True`. |
| 312 | |
| 313 | This method can only be called once and cannot be called after |
| 314 | :meth:`Future.set_result` or :meth:`Future.set_exception` have been |
| 315 | called. |
| 316 | |
| 317 | .. method:: set_result(result) |
| 318 | |
| 319 | Sets the result of the work associated with the :class:`Future` to |
| 320 | *result*. |
| 321 | |
| 322 | This method should only be used by :class:`Executor` implementations and |
| 323 | unit tests. |
| 324 | |
| 325 | .. method:: set_exception(exception) |
| 326 | |
| 327 | Sets the result of the work associated with the :class:`Future` to the |
| 328 | :class:`Exception` *exception*. |
| 329 | |
| 330 | This method should only be used by :class:`Executor` implementations and |
| 331 | unit tests. |
| 332 | |
| 333 | |
| 334 | Module Functions |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 335 | ---------------- |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 336 | |
| 337 | .. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) |
| 338 | |
| 339 | Wait for the :class:`Future` instances (possibly created by different |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 340 | :class:`Executor` instances) given by *fs* to complete. Returns a named |
| 341 | 2-tuple of sets. The first set, named ``done``, contains the futures that |
| 342 | completed (finished or were cancelled) before the wait completed. The second |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 343 | set, named ``not_done``, contains uncompleted futures. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 344 | |
| 345 | *timeout* can be used to control the maximum number of seconds to wait before |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 346 | returning. *timeout* can be an int or float. If *timeout* is not specified |
| 347 | or ``None``, there is no limit to the wait time. |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 348 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 349 | *return_when* indicates when this function should return. It must be one of |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 350 | the following constants: |
| 351 | |
Georg Brandl | 44ea77b | 2013-03-28 13:28:44 +0100 | [diff] [blame] | 352 | .. tabularcolumns:: |l|L| |
| 353 | |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 354 | +-----------------------------+----------------------------------------+ |
| 355 | | Constant | Description | |
| 356 | +=============================+========================================+ |
| 357 | | :const:`FIRST_COMPLETED` | The function will return when any | |
| 358 | | | future finishes or is cancelled. | |
| 359 | +-----------------------------+----------------------------------------+ |
| 360 | | :const:`FIRST_EXCEPTION` | The function will return when any | |
| 361 | | | future finishes by raising an | |
| 362 | | | exception. If no future raises an | |
| 363 | | | exception then it is equivalent to | |
| 364 | | | :const:`ALL_COMPLETED`. | |
| 365 | +-----------------------------+----------------------------------------+ |
| 366 | | :const:`ALL_COMPLETED` | The function will return when all | |
| 367 | | | futures finish or are cancelled. | |
| 368 | +-----------------------------+----------------------------------------+ |
Brian Quinlan | 81c4d36 | 2010-09-18 22:35:02 +0000 | [diff] [blame] | 369 | |
| 370 | .. function:: as_completed(fs, timeout=None) |
| 371 | |
Benjamin Peterson | c713fc7 | 2010-09-19 04:23:17 +0000 | [diff] [blame] | 372 | Returns an iterator over the :class:`Future` instances (possibly created by |
| 373 | different :class:`Executor` instances) given by *fs* that yields futures as |
Guido van Rossum | e6994ff | 2014-01-26 09:57:51 -0800 | [diff] [blame] | 374 | they complete (finished or were cancelled). Any futures given by *fs* that |
| 375 | are duplicated will be returned once. Any futures that completed |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 376 | before :func:`as_completed` is called will be yielded first. The returned |
Ezio Melotti | 7fa8222 | 2012-10-12 13:42:08 +0300 | [diff] [blame] | 377 | iterator raises a :exc:`TimeoutError` if :meth:`~iterator.__next__` is |
| 378 | called and the result isn't available after *timeout* seconds from the |
| 379 | original call to :func:`as_completed`. *timeout* can be an int or float. |
| 380 | If *timeout* is not specified or ``None``, there is no limit to the wait |
| 381 | time. |
Georg Brandl | 035cedb | 2010-09-19 09:31:09 +0000 | [diff] [blame] | 382 | |
| 383 | |
| 384 | .. seealso:: |
| 385 | |
| 386 | :pep:`3148` -- futures - execute computations asynchronously |
| 387 | The proposal which described this feature for inclusion in the Python |
| 388 | standard library. |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 389 | |
| 390 | |
| 391 | Exception classes |
| 392 | ----------------- |
| 393 | |
Georg Brandl | ce64ced | 2014-10-28 22:58:24 +0100 | [diff] [blame] | 394 | .. currentmodule:: concurrent.futures.process |
| 395 | |
Antoine Pitrou | dd69649 | 2011-06-08 17:21:55 +0200 | [diff] [blame] | 396 | .. exception:: BrokenProcessPool |
| 397 | |
| 398 | Derived from :exc:`RuntimeError`, this exception class is raised when |
| 399 | one of the workers of a :class:`ProcessPoolExecutor` has terminated |
| 400 | in a non-clean fashion (for example, if it was killed from the outside). |
| 401 | |
| 402 | .. versionadded:: 3.3 |
| 403 | |