Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 1 | :mod:`queue` --- A synchronized queue class |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 2 | =========================================== |
| 3 | |
Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 4 | .. module:: queue |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 5 | :synopsis: A synchronized queue class. |
| 6 | |
Raymond Hettinger | 1048094 | 2011-01-10 03:26:08 +0000 | [diff] [blame] | 7 | **Source code:** :source:`Lib/queue.py` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 8 | |
Raymond Hettinger | 4f707fd | 2011-01-10 19:54:11 +0000 | [diff] [blame] | 9 | -------------- |
| 10 | |
Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 11 | The :mod:`queue` module implements multi-producer, multi-consumer queues. |
Thomas Wouters | 89d996e | 2007-09-08 17:39:28 +0000 | [diff] [blame] | 12 | It is especially useful in threaded programming when information must be |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 13 | exchanged safely between multiple threads. The :class:`Queue` class in this |
| 14 | module implements all the required locking semantics. It depends on the |
Thomas Wouters | 89d996e | 2007-09-08 17:39:28 +0000 | [diff] [blame] | 15 | availability of thread support in Python; see the :mod:`threading` |
| 16 | module. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 17 | |
R David Murray | b98b37f | 2012-05-08 21:28:24 -0400 | [diff] [blame] | 18 | The module implements three types of queue, which differ only in the order in |
Serhiy Storchaka | 4ecfa45 | 2016-05-16 09:31:54 +0300 | [diff] [blame] | 19 | which the entries are retrieved. In a :abbr:`FIFO (first-in, first-out)` |
| 20 | queue, the first tasks added are the first retrieved. In a |
| 21 | :abbr:`LIFO (last-in, first-out)` queue, the most recently added entry is |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 22 | the first retrieved (operating like a stack). With a priority queue, |
| 23 | the entries are kept sorted (using the :mod:`heapq` module) and the |
| 24 | lowest valued entry is retrieved first. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 25 | |
Antoine Pitrou | 94e1696 | 2018-01-16 00:27:16 +0100 | [diff] [blame] | 26 | Internally, those three types of queues use locks to temporarily block |
| 27 | competing threads; however, they are not designed to handle reentrancy |
| 28 | within a thread. |
| 29 | |
| 30 | In addition, the module implements a "simple" |
| 31 | :abbr:`FIFO (first-in, first-out)` queue type where |
| 32 | specific implementations can provide additional guarantees |
| 33 | in exchange for the smaller functionality. |
Éric Araujo | 6e6cb8e | 2010-11-16 19:13:50 +0000 | [diff] [blame] | 34 | |
Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 35 | The :mod:`queue` module defines the following classes and exceptions: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 36 | |
Andrew M. Kuchling | 2b600e5 | 2010-02-26 13:35:56 +0000 | [diff] [blame] | 37 | .. class:: Queue(maxsize=0) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 38 | |
Serhiy Storchaka | 4ecfa45 | 2016-05-16 09:31:54 +0300 | [diff] [blame] | 39 | Constructor for a :abbr:`FIFO (first-in, first-out)` queue. *maxsize* is |
| 40 | an integer that sets the upperbound |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 41 | limit on the number of items that can be placed in the queue. Insertion will |
| 42 | block once this size has been reached, until queue items are consumed. If |
| 43 | *maxsize* is less than or equal to zero, the queue size is infinite. |
| 44 | |
Andrew M. Kuchling | 2b600e5 | 2010-02-26 13:35:56 +0000 | [diff] [blame] | 45 | .. class:: LifoQueue(maxsize=0) |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 46 | |
Serhiy Storchaka | 4ecfa45 | 2016-05-16 09:31:54 +0300 | [diff] [blame] | 47 | Constructor for a :abbr:`LIFO (last-in, first-out)` queue. *maxsize* is |
| 48 | an integer that sets the upperbound |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 49 | limit on the number of items that can be placed in the queue. Insertion will |
| 50 | block once this size has been reached, until queue items are consumed. If |
| 51 | *maxsize* is less than or equal to zero, the queue size is infinite. |
| 52 | |
Christian Heimes | 679db4a | 2008-01-18 09:56:22 +0000 | [diff] [blame] | 53 | |
Andrew M. Kuchling | 2b600e5 | 2010-02-26 13:35:56 +0000 | [diff] [blame] | 54 | .. class:: PriorityQueue(maxsize=0) |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 55 | |
| 56 | Constructor for a priority queue. *maxsize* is an integer that sets the upperbound |
| 57 | limit on the number of items that can be placed in the queue. Insertion will |
| 58 | block once this size has been reached, until queue items are consumed. If |
| 59 | *maxsize* is less than or equal to zero, the queue size is infinite. |
| 60 | |
| 61 | The lowest valued entries are retrieved first (the lowest valued entry is the |
| 62 | one returned by ``sorted(list(entries))[0]``). A typical pattern for entries |
| 63 | is a tuple in the form: ``(priority_number, data)``. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 64 | |
Raymond Hettinger | 0c3be96 | 2018-01-11 22:06:34 -0800 | [diff] [blame] | 65 | If the *data* elements are not comparable, the data can be wrapped in a class |
| 66 | that ignores the data item and only compares the priority number:: |
| 67 | |
| 68 | from dataclasses import dataclass, field |
| 69 | from typing import Any |
| 70 | |
| 71 | @dataclass(order=True) |
| 72 | class PrioritizedItem: |
| 73 | priority: int |
| 74 | item: Any=field(compare=False) |
Christian Heimes | 679db4a | 2008-01-18 09:56:22 +0000 | [diff] [blame] | 75 | |
Antoine Pitrou | 94e1696 | 2018-01-16 00:27:16 +0100 | [diff] [blame] | 76 | .. class:: SimpleQueue() |
| 77 | |
| 78 | Constructor for an unbounded :abbr:`FIFO (first-in, first-out)` queue. |
| 79 | Simple queues lack advanced functionality such as task tracking. |
| 80 | |
| 81 | .. versionadded:: 3.7 |
| 82 | |
| 83 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 84 | .. exception:: Empty |
| 85 | |
Serhiy Storchaka | 9e0ae53 | 2013-08-24 00:23:38 +0300 | [diff] [blame] | 86 | Exception raised when non-blocking :meth:`~Queue.get` (or |
| 87 | :meth:`~Queue.get_nowait`) is called |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 88 | on a :class:`Queue` object which is empty. |
| 89 | |
| 90 | |
| 91 | .. exception:: Full |
| 92 | |
Serhiy Storchaka | 9e0ae53 | 2013-08-24 00:23:38 +0300 | [diff] [blame] | 93 | Exception raised when non-blocking :meth:`~Queue.put` (or |
| 94 | :meth:`~Queue.put_nowait`) is called |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 95 | on a :class:`Queue` object which is full. |
| 96 | |
| 97 | |
| 98 | .. _queueobjects: |
| 99 | |
| 100 | Queue Objects |
| 101 | ------------- |
| 102 | |
Christian Heimes | 292d351 | 2008-02-03 16:51:08 +0000 | [diff] [blame] | 103 | Queue objects (:class:`Queue`, :class:`LifoQueue`, or :class:`PriorityQueue`) |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 104 | provide the public methods described below. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 105 | |
| 106 | |
| 107 | .. method:: Queue.qsize() |
| 108 | |
Guido van Rossum | 7736b5b | 2008-01-15 21:44:53 +0000 | [diff] [blame] | 109 | Return the approximate size of the queue. Note, qsize() > 0 doesn't |
| 110 | guarantee that a subsequent get() will not block, nor will qsize() < maxsize |
| 111 | guarantee that put() will not block. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 112 | |
| 113 | |
Raymond Hettinger | 47aa989 | 2009-03-07 14:07:37 +0000 | [diff] [blame] | 114 | .. method:: Queue.empty() |
| 115 | |
| 116 | Return ``True`` if the queue is empty, ``False`` otherwise. If empty() |
| 117 | returns ``True`` it doesn't guarantee that a subsequent call to put() |
| 118 | will not block. Similarly, if empty() returns ``False`` it doesn't |
| 119 | guarantee that a subsequent call to get() will not block. |
| 120 | |
| 121 | |
| 122 | .. method:: Queue.full() |
| 123 | |
| 124 | Return ``True`` if the queue is full, ``False`` otherwise. If full() |
| 125 | returns ``True`` it doesn't guarantee that a subsequent call to get() |
| 126 | will not block. Similarly, if full() returns ``False`` it doesn't |
| 127 | guarantee that a subsequent call to put() will not block. |
| 128 | |
| 129 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 130 | .. method:: Queue.put(item, block=True, timeout=None) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 131 | |
| 132 | Put *item* into the queue. If optional args *block* is true and *timeout* is |
Serhiy Storchaka | ecf41da | 2016-10-19 16:29:26 +0300 | [diff] [blame] | 133 | ``None`` (the default), block if necessary until a free slot is available. If |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 134 | *timeout* is a positive number, it blocks at most *timeout* seconds and raises |
| 135 | the :exc:`Full` exception if no free slot was available within that time. |
| 136 | Otherwise (*block* is false), put an item on the queue if a free slot is |
| 137 | immediately available, else raise the :exc:`Full` exception (*timeout* is |
| 138 | ignored in that case). |
| 139 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 140 | |
| 141 | .. method:: Queue.put_nowait(item) |
| 142 | |
| 143 | Equivalent to ``put(item, False)``. |
| 144 | |
| 145 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 146 | .. method:: Queue.get(block=True, timeout=None) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 147 | |
| 148 | Remove and return an item from the queue. If optional args *block* is true and |
Serhiy Storchaka | ecf41da | 2016-10-19 16:29:26 +0300 | [diff] [blame] | 149 | *timeout* is ``None`` (the default), block if necessary until an item is available. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 150 | If *timeout* is a positive number, it blocks at most *timeout* seconds and |
| 151 | raises the :exc:`Empty` exception if no item was available within that time. |
| 152 | Otherwise (*block* is false), return an item if one is immediately available, |
| 153 | else raise the :exc:`Empty` exception (*timeout* is ignored in that case). |
| 154 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 155 | |
| 156 | .. method:: Queue.get_nowait() |
| 157 | |
| 158 | Equivalent to ``get(False)``. |
| 159 | |
| 160 | Two methods are offered to support tracking whether enqueued tasks have been |
| 161 | fully processed by daemon consumer threads. |
| 162 | |
| 163 | |
| 164 | .. method:: Queue.task_done() |
| 165 | |
| 166 | Indicate that a formerly enqueued task is complete. Used by queue consumer |
| 167 | threads. For each :meth:`get` used to fetch a task, a subsequent call to |
| 168 | :meth:`task_done` tells the queue that the processing on the task is complete. |
| 169 | |
| 170 | If a :meth:`join` is currently blocking, it will resume when all items have been |
| 171 | processed (meaning that a :meth:`task_done` call was received for every item |
| 172 | that had been :meth:`put` into the queue). |
| 173 | |
| 174 | Raises a :exc:`ValueError` if called more times than there were items placed in |
| 175 | the queue. |
| 176 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 177 | |
| 178 | .. method:: Queue.join() |
| 179 | |
| 180 | Blocks until all items in the queue have been gotten and processed. |
| 181 | |
| 182 | The count of unfinished tasks goes up whenever an item is added to the queue. |
| 183 | The count goes down whenever a consumer thread calls :meth:`task_done` to |
| 184 | indicate that the item was retrieved and all work on it is complete. When the |
Raymond Hettinger | 28c013d | 2009-03-10 00:07:25 +0000 | [diff] [blame] | 185 | count of unfinished tasks drops to zero, :meth:`join` unblocks. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 186 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 187 | |
| 188 | Example of how to wait for enqueued tasks to be completed:: |
| 189 | |
Victor Stinner | de31134 | 2015-03-18 14:05:43 +0100 | [diff] [blame] | 190 | def worker(): |
| 191 | while True: |
| 192 | item = q.get() |
| 193 | if item is None: |
| 194 | break |
| 195 | do_work(item) |
| 196 | q.task_done() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 197 | |
Victor Stinner | de31134 | 2015-03-18 14:05:43 +0100 | [diff] [blame] | 198 | q = queue.Queue() |
| 199 | threads = [] |
| 200 | for i in range(num_worker_threads): |
| 201 | t = threading.Thread(target=worker) |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 202 | t.start() |
Victor Stinner | de31134 | 2015-03-18 14:05:43 +0100 | [diff] [blame] | 203 | threads.append(t) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 204 | |
Victor Stinner | de31134 | 2015-03-18 14:05:43 +0100 | [diff] [blame] | 205 | for item in source(): |
| 206 | q.put(item) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 207 | |
Victor Stinner | de31134 | 2015-03-18 14:05:43 +0100 | [diff] [blame] | 208 | # block until all tasks are done |
| 209 | q.join() |
| 210 | |
| 211 | # stop workers |
| 212 | for i in range(num_worker_threads): |
| 213 | q.put(None) |
| 214 | for t in threads: |
| 215 | t.join() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 216 | |
Antoine Pitrou | 696efdd | 2011-01-07 19:16:12 +0000 | [diff] [blame] | 217 | |
Antoine Pitrou | 94e1696 | 2018-01-16 00:27:16 +0100 | [diff] [blame] | 218 | SimpleQueue Objects |
| 219 | ------------------- |
| 220 | |
| 221 | :class:`SimpleQueue` objects provide the public methods described below. |
| 222 | |
| 223 | .. method:: SimpleQueue.qsize() |
| 224 | |
| 225 | Return the approximate size of the queue. Note, qsize() > 0 doesn't |
| 226 | guarantee that a subsequent get() will not block. |
| 227 | |
| 228 | |
| 229 | .. method:: SimpleQueue.empty() |
| 230 | |
| 231 | Return ``True`` if the queue is empty, ``False`` otherwise. If empty() |
| 232 | returns ``False`` it doesn't guarantee that a subsequent call to get() |
| 233 | will not block. |
| 234 | |
| 235 | |
| 236 | .. method:: SimpleQueue.put(item, block=True, timeout=None) |
| 237 | |
| 238 | Put *item* into the queue. The method never blocks and always succeeds |
| 239 | (except for potential low-level errors such as failure to allocate memory). |
| 240 | The optional args *block* and *timeout* are ignored and only provided |
| 241 | for compatibility with :meth:`Queue.put`. |
| 242 | |
| 243 | .. impl-detail:: |
| 244 | This method has a C implementation which is reentrant. That is, a |
| 245 | ``put()`` or ``get()`` call can be interrupted by another ``put()`` |
| 246 | call in the same thread without deadlocking or corrupting internal |
| 247 | state inside the queue. This makes it appropriate for use in |
| 248 | destructors such as ``__del__`` methods or :mod:`weakref` callbacks. |
| 249 | |
| 250 | |
| 251 | .. method:: SimpleQueue.put_nowait(item) |
| 252 | |
| 253 | Equivalent to ``put(item)``, provided for compatibility with |
| 254 | :meth:`Queue.put_nowait`. |
| 255 | |
| 256 | |
| 257 | .. method:: SimpleQueue.get(block=True, timeout=None) |
| 258 | |
| 259 | Remove and return an item from the queue. If optional args *block* is true and |
| 260 | *timeout* is ``None`` (the default), block if necessary until an item is available. |
| 261 | If *timeout* is a positive number, it blocks at most *timeout* seconds and |
| 262 | raises the :exc:`Empty` exception if no item was available within that time. |
| 263 | Otherwise (*block* is false), return an item if one is immediately available, |
| 264 | else raise the :exc:`Empty` exception (*timeout* is ignored in that case). |
| 265 | |
| 266 | |
| 267 | .. method:: SimpleQueue.get_nowait() |
| 268 | |
| 269 | Equivalent to ``get(False)``. |
| 270 | |
| 271 | |
Antoine Pitrou | 696efdd | 2011-01-07 19:16:12 +0000 | [diff] [blame] | 272 | .. seealso:: |
| 273 | |
| 274 | Class :class:`multiprocessing.Queue` |
| 275 | A queue class for use in a multi-processing (rather than multi-threading) |
| 276 | context. |
| 277 | |
Georg Brandl | 2f2a9f7 | 2011-01-07 20:58:25 +0000 | [diff] [blame] | 278 | :class:`collections.deque` is an alternative implementation of unbounded |
Serhiy Storchaka | 9e0ae53 | 2013-08-24 00:23:38 +0300 | [diff] [blame] | 279 | queues with fast atomic :meth:`~collections.deque.append` and |
| 280 | :meth:`~collections.deque.popleft` operations that do not require locking. |