blob: e5d13f37f3c2feff8fa82b26262d072281cba1c8 [file] [log] [blame]
Antoine Pitrou64a467d2010-12-12 20:34:49 +00001:mod:`concurrent.futures` --- Launching parallel tasks
2======================================================
Brian Quinlan81c4d362010-09-18 22:35:02 +00003
4.. module:: concurrent.futures
5 :synopsis: Execute computations concurrently using threads or processes.
6
Raymond Hettingera1993682011-01-27 01:20:32 +00007**Source code:** :source:`Lib/concurrent/futures/thread.py`
8and :source:`Lib/concurrent/futures/process.py`
9
Georg Brandl035cedb2010-09-19 09:31:09 +000010.. versionadded:: 3.2
11
Raymond Hettingera1993682011-01-27 01:20:32 +000012--------------
13
Brian Quinlan81c4d362010-09-18 22:35:02 +000014The :mod:`concurrent.futures` module provides a high-level interface for
15asynchronously executing callables.
16
Benjamin Petersonc713fc72010-09-19 04:23:17 +000017The asynchronous execution can be be performed with threads, using
Georg Brandlfb1720b2010-12-09 18:08:43 +000018:class:`ThreadPoolExecutor`, or separate processes, using
Georg Brandl035cedb2010-09-19 09:31:09 +000019:class:`ProcessPoolExecutor`. Both implement the same interface, which is
Brian Quinlan81c4d362010-09-18 22:35:02 +000020defined by the abstract :class:`Executor` class.
21
Georg Brandl035cedb2010-09-19 09:31:09 +000022
Brian Quinlan81c4d362010-09-18 22:35:02 +000023Executor Objects
Georg Brandl035cedb2010-09-19 09:31:09 +000024----------------
Brian Quinlan81c4d362010-09-18 22:35:02 +000025
Benjamin Petersonc713fc72010-09-19 04:23:17 +000026.. class:: Executor
Brian Quinlan81c4d362010-09-18 22:35:02 +000027
Georg Brandl035cedb2010-09-19 09:31:09 +000028 An abstract class that provides methods to execute calls asynchronously. It
Benjamin Petersonc713fc72010-09-19 04:23:17 +000029 should not be used directly, but through its concrete subclasses.
Brian Quinlan81c4d362010-09-18 22:35:02 +000030
31 .. method:: submit(fn, *args, **kwargs)
32
Benjamin Petersonc713fc72010-09-19 04:23:17 +000033 Schedules the callable, *fn*, to be executed as ``fn(*args **kwargs)``
34 and returns a :class:`Future` object representing the execution of the
Georg Brandl035cedb2010-09-19 09:31:09 +000035 callable. ::
Brian Quinlan81c4d362010-09-18 22:35:02 +000036
Georg Brandl035cedb2010-09-19 09:31:09 +000037 with ThreadPoolExecutor(max_workers=1) as executor:
38 future = executor.submit(pow, 323, 1235)
39 print(future.result())
Brian Quinlan81c4d362010-09-18 22:35:02 +000040
41 .. method:: map(func, *iterables, timeout=None)
42
Benjamin Petersonc713fc72010-09-19 04:23:17 +000043 Equivalent to ``map(func, *iterables)`` except *func* is executed
Georg Brandl035cedb2010-09-19 09:31:09 +000044 asynchronously and several calls to *func* may be made concurrently. The
Brian Quinlan81c4d362010-09-18 22:35:02 +000045 returned iterator raises a :exc:`TimeoutError` if :meth:`__next__()` is
46 called and the result isn't available after *timeout* seconds from the
Benjamin Petersonc713fc72010-09-19 04:23:17 +000047 original call to :meth:`Executor.map`. *timeout* can be an int or a
Georg Brandl035cedb2010-09-19 09:31:09 +000048 float. If *timeout* is not specified or ``None``, there is no limit to
49 the wait time. If a call raises an exception, then that exception will
50 be raised when its value is retrieved from the iterator.
Brian Quinlan81c4d362010-09-18 22:35:02 +000051
52 .. method:: shutdown(wait=True)
53
54 Signal the executor that it should free any resources that it is using
Georg Brandl035cedb2010-09-19 09:31:09 +000055 when the currently pending futures are done executing. Calls to
Brian Quinlan81c4d362010-09-18 22:35:02 +000056 :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will
57 raise :exc:`RuntimeError`.
58
Benjamin Petersonc713fc72010-09-19 04:23:17 +000059 If *wait* is ``True`` then this method will not return until all the
Brian Quinlan81c4d362010-09-18 22:35:02 +000060 pending futures are done executing and the resources associated with the
Georg Brandl035cedb2010-09-19 09:31:09 +000061 executor have been freed. If *wait* is ``False`` then this method will
Benjamin Petersonc713fc72010-09-19 04:23:17 +000062 return immediately and the resources associated with the executor will be
Georg Brandl035cedb2010-09-19 09:31:09 +000063 freed when all pending futures are done executing. Regardless of the
Brian Quinlan81c4d362010-09-18 22:35:02 +000064 value of *wait*, the entire Python program will not exit until all
65 pending futures are done executing.
66
Georg Brandl035cedb2010-09-19 09:31:09 +000067 You can avoid having to call this method explicitly if you use the
68 :keyword:`with` statement, which will shutdown the :class:`Executor`
69 (waiting as if :meth:`Executor.shutdown` were called with *wait* set to
70 ``True``)::
Brian Quinlan81c4d362010-09-18 22:35:02 +000071
Georg Brandl035cedb2010-09-19 09:31:09 +000072 import shutil
73 with ThreadPoolExecutor(max_workers=4) as e:
74 e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
75 e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
76 e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
77 e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
Brian Quinlan81c4d362010-09-18 22:35:02 +000078
Brian Quinlan81c4d362010-09-18 22:35:02 +000079
80ThreadPoolExecutor
Georg Brandl035cedb2010-09-19 09:31:09 +000081------------------
Brian Quinlan81c4d362010-09-18 22:35:02 +000082
Benjamin Petersonc713fc72010-09-19 04:23:17 +000083:class:`ThreadPoolExecutor` is a :class:`Executor` subclass that uses a pool of
84threads to execute calls asynchronously.
Brian Quinlan81c4d362010-09-18 22:35:02 +000085
Benjamin Petersonc713fc72010-09-19 04:23:17 +000086Deadlocks can occur when the callable associated with a :class:`Future` waits on
Georg Brandl035cedb2010-09-19 09:31:09 +000087the results of another :class:`Future`. For example::
Brian Quinlan81c4d362010-09-18 22:35:02 +000088
Georg Brandl035cedb2010-09-19 09:31:09 +000089 import time
90 def wait_on_b():
91 time.sleep(5)
92 print(b.result()) # b will never complete because it is waiting on a.
93 return 5
Brian Quinlan81c4d362010-09-18 22:35:02 +000094
Georg Brandl035cedb2010-09-19 09:31:09 +000095 def wait_on_a():
96 time.sleep(5)
97 print(a.result()) # a will never complete because it is waiting on b.
98 return 6
Brian Quinlan81c4d362010-09-18 22:35:02 +000099
100
Georg Brandl035cedb2010-09-19 09:31:09 +0000101 executor = ThreadPoolExecutor(max_workers=2)
102 a = executor.submit(wait_on_b)
103 b = executor.submit(wait_on_a)
Brian Quinlan81c4d362010-09-18 22:35:02 +0000104
Georg Brandl035cedb2010-09-19 09:31:09 +0000105And::
Brian Quinlan81c4d362010-09-18 22:35:02 +0000106
Georg Brandl035cedb2010-09-19 09:31:09 +0000107 def wait_on_future():
108 f = executor.submit(pow, 5, 2)
109 # This will never complete because there is only one worker thread and
110 # it is executing this function.
111 print(f.result())
Brian Quinlan81c4d362010-09-18 22:35:02 +0000112
Georg Brandl035cedb2010-09-19 09:31:09 +0000113 executor = ThreadPoolExecutor(max_workers=1)
114 executor.submit(wait_on_future)
Brian Quinlan81c4d362010-09-18 22:35:02 +0000115
116
117.. class:: ThreadPoolExecutor(max_workers)
118
119 An :class:`Executor` subclass that uses a pool of at most *max_workers*
120 threads to execute calls asynchronously.
121
Georg Brandl035cedb2010-09-19 09:31:09 +0000122
Brian Quinlan81c4d362010-09-18 22:35:02 +0000123.. _threadpoolexecutor-example:
124
125ThreadPoolExecutor Example
Georg Brandl035cedb2010-09-19 09:31:09 +0000126~~~~~~~~~~~~~~~~~~~~~~~~~~
Brian Quinlan81c4d362010-09-18 22:35:02 +0000127::
128
Georg Brandl035cedb2010-09-19 09:31:09 +0000129 import concurrent.futures
130 import urllib.request
Brian Quinlan81c4d362010-09-18 22:35:02 +0000131
Georg Brandl035cedb2010-09-19 09:31:09 +0000132 URLS = ['http://www.foxnews.com/',
133 'http://www.cnn.com/',
134 'http://europe.wsj.com/',
135 'http://www.bbc.co.uk/',
136 'http://some-made-up-domain.com/']
Brian Quinlan81c4d362010-09-18 22:35:02 +0000137
Georg Brandl035cedb2010-09-19 09:31:09 +0000138 def load_url(url, timeout):
139 return urllib.request.urlopen(url, timeout=timeout).read()
Brian Quinlan81c4d362010-09-18 22:35:02 +0000140
Georg Brandl035cedb2010-09-19 09:31:09 +0000141 with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
142 future_to_url = dict((executor.submit(load_url, url, 60), url)
143 for url in URLS)
Brian Quinlan81c4d362010-09-18 22:35:02 +0000144
Georg Brandl035cedb2010-09-19 09:31:09 +0000145 for future in concurrent.futures.as_completed(future_to_url):
146 url = future_to_url[future]
147 if future.exception() is not None:
148 print('%r generated an exception: %s' % (url,
149 future.exception()))
150 else:
151 print('%r page is %d bytes' % (url, len(future.result())))
Brian Quinlan81c4d362010-09-18 22:35:02 +0000152
153
154ProcessPoolExecutor
Georg Brandl035cedb2010-09-19 09:31:09 +0000155-------------------
Brian Quinlan81c4d362010-09-18 22:35:02 +0000156
157The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that
158uses a pool of processes to execute calls asynchronously.
159:class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which
160allows it to side-step the :term:`Global Interpreter Lock` but also means that
161only picklable objects can be executed and returned.
162
163Calling :class:`Executor` or :class:`Future` methods from a callable submitted
164to a :class:`ProcessPoolExecutor` will result in deadlock.
165
166.. class:: ProcessPoolExecutor(max_workers=None)
167
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000168 An :class:`Executor` subclass that executes calls asynchronously using a pool
Georg Brandl035cedb2010-09-19 09:31:09 +0000169 of at most *max_workers* processes. If *max_workers* is ``None`` or not
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000170 given, it will default to the number of processors on the machine.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000171
Georg Brandl035cedb2010-09-19 09:31:09 +0000172
Brian Quinlan81c4d362010-09-18 22:35:02 +0000173.. _processpoolexecutor-example:
174
175ProcessPoolExecutor Example
Georg Brandl035cedb2010-09-19 09:31:09 +0000176~~~~~~~~~~~~~~~~~~~~~~~~~~~
Brian Quinlan81c4d362010-09-18 22:35:02 +0000177::
178
Georg Brandl035cedb2010-09-19 09:31:09 +0000179 import concurrent.futures
180 import math
Brian Quinlan81c4d362010-09-18 22:35:02 +0000181
Georg Brandl035cedb2010-09-19 09:31:09 +0000182 PRIMES = [
183 112272535095293,
184 112582705942171,
185 112272535095293,
186 115280095190773,
187 115797848077099,
188 1099726899285419]
Brian Quinlan81c4d362010-09-18 22:35:02 +0000189
Georg Brandl035cedb2010-09-19 09:31:09 +0000190 def is_prime(n):
191 if n % 2 == 0:
192 return False
Brian Quinlan81c4d362010-09-18 22:35:02 +0000193
Georg Brandl035cedb2010-09-19 09:31:09 +0000194 sqrt_n = int(math.floor(math.sqrt(n)))
195 for i in range(3, sqrt_n + 1, 2):
196 if n % i == 0:
197 return False
198 return True
Brian Quinlan81c4d362010-09-18 22:35:02 +0000199
Georg Brandl035cedb2010-09-19 09:31:09 +0000200 def main():
201 with concurrent.futures.ProcessPoolExecutor() as executor:
202 for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
203 print('%d is prime: %s' % (number, prime))
Brian Quinlan81c4d362010-09-18 22:35:02 +0000204
Georg Brandl035cedb2010-09-19 09:31:09 +0000205 if __name__ == '__main__':
206 main()
207
Brian Quinlan81c4d362010-09-18 22:35:02 +0000208
209Future Objects
Georg Brandl035cedb2010-09-19 09:31:09 +0000210--------------
Brian Quinlan81c4d362010-09-18 22:35:02 +0000211
Georg Brandl6faee4e2010-09-21 14:48:28 +0000212The :class:`Future` class encapsulates the asynchronous execution of a callable.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000213:class:`Future` instances are created by :meth:`Executor.submit`.
214
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000215.. class:: Future
Brian Quinlan81c4d362010-09-18 22:35:02 +0000216
Georg Brandl6faee4e2010-09-21 14:48:28 +0000217 Encapsulates the asynchronous execution of a callable. :class:`Future`
Brian Quinlan81c4d362010-09-18 22:35:02 +0000218 instances are created by :meth:`Executor.submit` and should not be created
219 directly except for testing.
220
221 .. method:: cancel()
222
Georg Brandl035cedb2010-09-19 09:31:09 +0000223 Attempt to cancel the call. If the call is currently being executed and
Eric Smith7b5011b2011-02-01 21:31:22 +0000224 cannot be cancelled then the method will return ``False``, otherwise the
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000225 call will be cancelled and the method will return ``True``.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000226
227 .. method:: cancelled()
228
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000229 Return ``True`` if the call was successfully cancelled.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000230
231 .. method:: running()
232
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000233 Return ``True`` if the call is currently being executed and cannot be
Brian Quinlan81c4d362010-09-18 22:35:02 +0000234 cancelled.
235
236 .. method:: done()
237
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000238 Return ``True`` if the call was successfully cancelled or finished
239 running.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000240
241 .. method:: result(timeout=None)
242
243 Return the value returned by the call. If the call hasn't yet completed
Georg Brandl035cedb2010-09-19 09:31:09 +0000244 then this method will wait up to *timeout* seconds. If the call hasn't
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000245 completed in *timeout* seconds, then a :exc:`TimeoutError` will be
Georg Brandl035cedb2010-09-19 09:31:09 +0000246 raised. *timeout* can be an int or float. If *timeout* is not specified
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000247 or ``None``, there is no limit to the wait time.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000248
249 If the future is cancelled before completing then :exc:`CancelledError`
250 will be raised.
251
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000252 If the call raised, this method will raise the same exception.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000253
254 .. method:: exception(timeout=None)
255
Georg Brandl035cedb2010-09-19 09:31:09 +0000256 Return the exception raised by the call. If the call hasn't yet
257 completed then this method will wait up to *timeout* seconds. If the
258 call hasn't completed in *timeout* seconds, then a :exc:`TimeoutError`
259 will be raised. *timeout* can be an int or float. If *timeout* is not
260 specified or ``None``, there is no limit to the wait time.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000261
262 If the future is cancelled before completing then :exc:`CancelledError`
263 will be raised.
264
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000265 If the call completed without raising, ``None`` is returned.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000266
267 .. method:: add_done_callback(fn)
268
Georg Brandl035cedb2010-09-19 09:31:09 +0000269 Attaches the callable *fn* to the future. *fn* will be called, with the
Brian Quinlan81c4d362010-09-18 22:35:02 +0000270 future as its only argument, when the future is cancelled or finishes
271 running.
272
273 Added callables are called in the order that they were added and are
Georg Brandl035cedb2010-09-19 09:31:09 +0000274 always called in a thread belonging to the process that added them. If
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000275 the callable raises a :exc:`Exception` subclass, it will be logged and
Georg Brandl035cedb2010-09-19 09:31:09 +0000276 ignored. If the callable raises a :exc:`BaseException` subclass, the
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000277 behavior is undefined.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000278
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000279 If the future has already completed or been cancelled, *fn* will be
Brian Quinlan81c4d362010-09-18 22:35:02 +0000280 called immediately.
281
282 The following :class:`Future` methods are meant for use in unit tests and
283 :class:`Executor` implementations.
284
285 .. method:: set_running_or_notify_cancel()
286
287 This method should only be called by :class:`Executor` implementations
Georg Brandl035cedb2010-09-19 09:31:09 +0000288 before executing the work associated with the :class:`Future` and by unit
289 tests.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000290
Senthil Kumaran916bd382010-10-15 12:55:19 +0000291 If the method returns ``False`` then the :class:`Future` was cancelled,
Georg Brandl035cedb2010-09-19 09:31:09 +0000292 i.e. :meth:`Future.cancel` was called and returned `True`. Any threads
293 waiting on the :class:`Future` completing (i.e. through
294 :func:`as_completed` or :func:`wait`) will be woken up.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000295
Senthil Kumaran916bd382010-10-15 12:55:19 +0000296 If the method returns ``True`` then the :class:`Future` was not cancelled
Georg Brandl035cedb2010-09-19 09:31:09 +0000297 and has been put in the running state, i.e. calls to
Brian Quinlan81c4d362010-09-18 22:35:02 +0000298 :meth:`Future.running` will return `True`.
299
300 This method can only be called once and cannot be called after
301 :meth:`Future.set_result` or :meth:`Future.set_exception` have been
302 called.
303
304 .. method:: set_result(result)
305
306 Sets the result of the work associated with the :class:`Future` to
307 *result*.
308
309 This method should only be used by :class:`Executor` implementations and
310 unit tests.
311
312 .. method:: set_exception(exception)
313
314 Sets the result of the work associated with the :class:`Future` to the
315 :class:`Exception` *exception*.
316
317 This method should only be used by :class:`Executor` implementations and
318 unit tests.
319
320
321Module Functions
Georg Brandl035cedb2010-09-19 09:31:09 +0000322----------------
Brian Quinlan81c4d362010-09-18 22:35:02 +0000323
324.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED)
325
326 Wait for the :class:`Future` instances (possibly created by different
Georg Brandl035cedb2010-09-19 09:31:09 +0000327 :class:`Executor` instances) given by *fs* to complete. Returns a named
328 2-tuple of sets. The first set, named ``done``, contains the futures that
329 completed (finished or were cancelled) before the wait completed. The second
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000330 set, named ``not_done``, contains uncompleted futures.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000331
332 *timeout* can be used to control the maximum number of seconds to wait before
Georg Brandl035cedb2010-09-19 09:31:09 +0000333 returning. *timeout* can be an int or float. If *timeout* is not specified
334 or ``None``, there is no limit to the wait time.
Brian Quinlan81c4d362010-09-18 22:35:02 +0000335
Georg Brandl035cedb2010-09-19 09:31:09 +0000336 *return_when* indicates when this function should return. It must be one of
Brian Quinlan81c4d362010-09-18 22:35:02 +0000337 the following constants:
338
Georg Brandl035cedb2010-09-19 09:31:09 +0000339 +-----------------------------+----------------------------------------+
340 | Constant | Description |
341 +=============================+========================================+
342 | :const:`FIRST_COMPLETED` | The function will return when any |
343 | | future finishes or is cancelled. |
344 +-----------------------------+----------------------------------------+
345 | :const:`FIRST_EXCEPTION` | The function will return when any |
346 | | future finishes by raising an |
347 | | exception. If no future raises an |
348 | | exception then it is equivalent to |
349 | | :const:`ALL_COMPLETED`. |
350 +-----------------------------+----------------------------------------+
351 | :const:`ALL_COMPLETED` | The function will return when all |
352 | | futures finish or are cancelled. |
353 +-----------------------------+----------------------------------------+
Brian Quinlan81c4d362010-09-18 22:35:02 +0000354
355.. function:: as_completed(fs, timeout=None)
356
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000357 Returns an iterator over the :class:`Future` instances (possibly created by
358 different :class:`Executor` instances) given by *fs* that yields futures as
Georg Brandl035cedb2010-09-19 09:31:09 +0000359 they complete (finished or were cancelled). Any futures that completed
360 before :func:`as_completed` is called will be yielded first. The returned
361 iterator raises a :exc:`TimeoutError` if :meth:`__next__` is called and the
362 result isn't available after *timeout* seconds from the original call to
363 :func:`as_completed`. *timeout* can be an int or float. If *timeout* is not
Benjamin Petersonc713fc72010-09-19 04:23:17 +0000364 specified or ``None``, there is no limit to the wait time.
Georg Brandl035cedb2010-09-19 09:31:09 +0000365
366
367.. seealso::
368
369 :pep:`3148` -- futures - execute computations asynchronously
370 The proposal which described this feature for inclusion in the Python
371 standard library.