Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 1 | :mod:`queue` --- A synchronized queue class |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 2 | =========================================== |
| 3 | |
Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 4 | .. module:: queue |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 5 | :synopsis: A synchronized queue class. |
| 6 | |
Raymond Hettinger | 1048094 | 2011-01-10 03:26:08 +0000 | [diff] [blame] | 7 | **Source code:** :source:`Lib/queue.py` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 8 | |
Raymond Hettinger | 4f707fd | 2011-01-10 19:54:11 +0000 | [diff] [blame] | 9 | -------------- |
| 10 | |
Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 11 | The :mod:`queue` module implements multi-producer, multi-consumer queues. |
Thomas Wouters | 89d996e | 2007-09-08 17:39:28 +0000 | [diff] [blame] | 12 | It is especially useful in threaded programming when information must be |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 13 | exchanged safely between multiple threads. The :class:`Queue` class in this |
| 14 | module implements all the required locking semantics. It depends on the |
Thomas Wouters | 89d996e | 2007-09-08 17:39:28 +0000 | [diff] [blame] | 15 | availability of thread support in Python; see the :mod:`threading` |
| 16 | module. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 17 | |
R David Murray | b98b37f | 2012-05-08 21:28:24 -0400 | [diff] [blame] | 18 | The module implements three types of queue, which differ only in the order in |
| 19 | which the entries are retrieved. In a FIFO queue, the first tasks added are |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 20 | the first retrieved. In a LIFO queue, the most recently added entry is |
| 21 | the first retrieved (operating like a stack). With a priority queue, |
| 22 | the entries are kept sorted (using the :mod:`heapq` module) and the |
| 23 | lowest valued entry is retrieved first. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 24 | |
Éric Araujo | 6e6cb8e | 2010-11-16 19:13:50 +0000 | [diff] [blame] | 25 | |
Alexandre Vassalotti | f260e44 | 2008-05-11 19:59:59 +0000 | [diff] [blame] | 26 | The :mod:`queue` module defines the following classes and exceptions: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 27 | |
Andrew M. Kuchling | 2b600e5 | 2010-02-26 13:35:56 +0000 | [diff] [blame] | 28 | .. class:: Queue(maxsize=0) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 29 | |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 30 | Constructor for a FIFO queue. *maxsize* is an integer that sets the upperbound |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 31 | limit on the number of items that can be placed in the queue. Insertion will |
| 32 | block once this size has been reached, until queue items are consumed. If |
| 33 | *maxsize* is less than or equal to zero, the queue size is infinite. |
| 34 | |
Andrew M. Kuchling | 2b600e5 | 2010-02-26 13:35:56 +0000 | [diff] [blame] | 35 | .. class:: LifoQueue(maxsize=0) |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 36 | |
| 37 | Constructor for a LIFO queue. *maxsize* is an integer that sets the upperbound |
| 38 | limit on the number of items that can be placed in the queue. Insertion will |
| 39 | block once this size has been reached, until queue items are consumed. If |
| 40 | *maxsize* is less than or equal to zero, the queue size is infinite. |
| 41 | |
Christian Heimes | 679db4a | 2008-01-18 09:56:22 +0000 | [diff] [blame] | 42 | |
Andrew M. Kuchling | 2b600e5 | 2010-02-26 13:35:56 +0000 | [diff] [blame] | 43 | .. class:: PriorityQueue(maxsize=0) |
Raymond Hettinger | 3564146 | 2008-01-17 00:13:27 +0000 | [diff] [blame] | 44 | |
| 45 | Constructor for a priority queue. *maxsize* is an integer that sets the upperbound |
| 46 | limit on the number of items that can be placed in the queue. Insertion will |
| 47 | block once this size has been reached, until queue items are consumed. If |
| 48 | *maxsize* is less than or equal to zero, the queue size is infinite. |
| 49 | |
| 50 | The lowest valued entries are retrieved first (the lowest valued entry is the |
| 51 | one returned by ``sorted(list(entries))[0]``). A typical pattern for entries |
| 52 | is a tuple in the form: ``(priority_number, data)``. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 53 | |
Christian Heimes | 679db4a | 2008-01-18 09:56:22 +0000 | [diff] [blame] | 54 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 55 | .. exception:: Empty |
| 56 | |
| 57 | Exception raised when non-blocking :meth:`get` (or :meth:`get_nowait`) is called |
| 58 | on a :class:`Queue` object which is empty. |
| 59 | |
| 60 | |
| 61 | .. exception:: Full |
| 62 | |
| 63 | Exception raised when non-blocking :meth:`put` (or :meth:`put_nowait`) is called |
| 64 | on a :class:`Queue` object which is full. |
| 65 | |
| 66 | |
| 67 | .. _queueobjects: |
| 68 | |
| 69 | Queue Objects |
| 70 | ------------- |
| 71 | |
Christian Heimes | 292d351 | 2008-02-03 16:51:08 +0000 | [diff] [blame] | 72 | Queue objects (:class:`Queue`, :class:`LifoQueue`, or :class:`PriorityQueue`) |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 73 | provide the public methods described below. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 74 | |
| 75 | |
| 76 | .. method:: Queue.qsize() |
| 77 | |
Guido van Rossum | 7736b5b | 2008-01-15 21:44:53 +0000 | [diff] [blame] | 78 | Return the approximate size of the queue. Note, qsize() > 0 doesn't |
| 79 | guarantee that a subsequent get() will not block, nor will qsize() < maxsize |
| 80 | guarantee that put() will not block. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 81 | |
| 82 | |
Raymond Hettinger | 47aa989 | 2009-03-07 14:07:37 +0000 | [diff] [blame] | 83 | .. method:: Queue.empty() |
| 84 | |
| 85 | Return ``True`` if the queue is empty, ``False`` otherwise. If empty() |
| 86 | returns ``True`` it doesn't guarantee that a subsequent call to put() |
| 87 | will not block. Similarly, if empty() returns ``False`` it doesn't |
| 88 | guarantee that a subsequent call to get() will not block. |
| 89 | |
| 90 | |
| 91 | .. method:: Queue.full() |
| 92 | |
| 93 | Return ``True`` if the queue is full, ``False`` otherwise. If full() |
| 94 | returns ``True`` it doesn't guarantee that a subsequent call to get() |
| 95 | will not block. Similarly, if full() returns ``False`` it doesn't |
| 96 | guarantee that a subsequent call to put() will not block. |
| 97 | |
| 98 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 99 | .. method:: Queue.put(item, block=True, timeout=None) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 100 | |
| 101 | Put *item* into the queue. If optional args *block* is true and *timeout* is |
| 102 | None (the default), block if necessary until a free slot is available. If |
| 103 | *timeout* is a positive number, it blocks at most *timeout* seconds and raises |
| 104 | the :exc:`Full` exception if no free slot was available within that time. |
| 105 | Otherwise (*block* is false), put an item on the queue if a free slot is |
| 106 | immediately available, else raise the :exc:`Full` exception (*timeout* is |
| 107 | ignored in that case). |
| 108 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 109 | |
| 110 | .. method:: Queue.put_nowait(item) |
| 111 | |
| 112 | Equivalent to ``put(item, False)``. |
| 113 | |
| 114 | |
Georg Brandl | 1824415 | 2009-09-02 20:34:52 +0000 | [diff] [blame] | 115 | .. method:: Queue.get(block=True, timeout=None) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 116 | |
| 117 | Remove and return an item from the queue. If optional args *block* is true and |
| 118 | *timeout* is None (the default), block if necessary until an item is available. |
| 119 | If *timeout* is a positive number, it blocks at most *timeout* seconds and |
| 120 | raises the :exc:`Empty` exception if no item was available within that time. |
| 121 | Otherwise (*block* is false), return an item if one is immediately available, |
| 122 | else raise the :exc:`Empty` exception (*timeout* is ignored in that case). |
| 123 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 124 | |
| 125 | .. method:: Queue.get_nowait() |
| 126 | |
| 127 | Equivalent to ``get(False)``. |
| 128 | |
| 129 | Two methods are offered to support tracking whether enqueued tasks have been |
| 130 | fully processed by daemon consumer threads. |
| 131 | |
| 132 | |
| 133 | .. method:: Queue.task_done() |
| 134 | |
| 135 | Indicate that a formerly enqueued task is complete. Used by queue consumer |
| 136 | threads. For each :meth:`get` used to fetch a task, a subsequent call to |
| 137 | :meth:`task_done` tells the queue that the processing on the task is complete. |
| 138 | |
| 139 | If a :meth:`join` is currently blocking, it will resume when all items have been |
| 140 | processed (meaning that a :meth:`task_done` call was received for every item |
| 141 | that had been :meth:`put` into the queue). |
| 142 | |
| 143 | Raises a :exc:`ValueError` if called more times than there were items placed in |
| 144 | the queue. |
| 145 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 146 | |
| 147 | .. method:: Queue.join() |
| 148 | |
| 149 | Blocks until all items in the queue have been gotten and processed. |
| 150 | |
| 151 | The count of unfinished tasks goes up whenever an item is added to the queue. |
| 152 | The count goes down whenever a consumer thread calls :meth:`task_done` to |
| 153 | indicate that the item was retrieved and all work on it is complete. When the |
Raymond Hettinger | 28c013d | 2009-03-10 00:07:25 +0000 | [diff] [blame] | 154 | count of unfinished tasks drops to zero, :meth:`join` unblocks. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 155 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 156 | |
| 157 | Example of how to wait for enqueued tasks to be completed:: |
| 158 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 159 | def worker(): |
| 160 | while True: |
| 161 | item = q.get() |
| 162 | do_work(item) |
| 163 | q.task_done() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 164 | |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 165 | q = Queue() |
| 166 | for i in range(num_worker_threads): |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 167 | t = Thread(target=worker) |
Benjamin Peterson | e8fcbf6 | 2009-01-30 02:29:43 +0000 | [diff] [blame] | 168 | t.daemon = True |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 169 | t.start() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 170 | |
| 171 | for item in source(): |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 172 | q.put(item) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 173 | |
| 174 | q.join() # block until all tasks are done |
| 175 | |
Antoine Pitrou | 696efdd | 2011-01-07 19:16:12 +0000 | [diff] [blame] | 176 | |
| 177 | .. seealso:: |
| 178 | |
| 179 | Class :class:`multiprocessing.Queue` |
| 180 | A queue class for use in a multi-processing (rather than multi-threading) |
| 181 | context. |
| 182 | |
Georg Brandl | 2f2a9f7 | 2011-01-07 20:58:25 +0000 | [diff] [blame] | 183 | :class:`collections.deque` is an alternative implementation of unbounded |
Raymond Hettinger | fc90213 | 2011-01-07 20:33:09 +0000 | [diff] [blame] | 184 | queues with fast atomic :func:`append` and :func:`popleft` operations that |
| 185 | do not require locking. |
| 186 | |