Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`zlib` --- Compression compatible with :program:`gzip` |
| 2 | =========================================================== |
| 3 | |
| 4 | .. module:: zlib |
Georg Brandl | 7f01a13 | 2009-09-16 15:58:14 +0000 | [diff] [blame] | 5 | :synopsis: Low-level interface to compression and decompression routines |
| 6 | compatible with gzip. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 7 | |
| 8 | |
| 9 | For applications that require data compression, the functions in this module |
| 10 | allow compression and decompression, using the zlib library. The zlib library |
| 11 | has its own home page at http://www.zlib.net. There are known |
| 12 | incompatibilities between the Python module and versions of the zlib library |
| 13 | earlier than 1.1.3; 1.1.3 has a security vulnerability, so we recommend using |
| 14 | 1.1.4 or later. |
| 15 | |
| 16 | zlib's functions have many options and often need to be used in a particular |
| 17 | order. This documentation doesn't attempt to cover all of the permutations; |
| 18 | consult the zlib manual at http://www.zlib.net/manual.html for authoritative |
| 19 | information. |
| 20 | |
Éric Araujo | f2fbb9c | 2012-01-16 16:55:55 +0100 | [diff] [blame] | 21 | For reading and writing ``.gz`` files see the :mod:`gzip` module. |
Guido van Rossum | 7767711 | 2007-11-05 19:43:04 +0000 | [diff] [blame] | 22 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 23 | The available exception and functions in this module are: |
| 24 | |
| 25 | |
| 26 | .. exception:: error |
| 27 | |
| 28 | Exception raised on compression and decompression errors. |
| 29 | |
| 30 | |
Benjamin Peterson | 058e31e | 2009-01-16 03:54:08 +0000 | [diff] [blame] | 31 | .. function:: adler32(data[, value]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 32 | |
Serhiy Storchaka | d65c949 | 2015-11-02 14:10:23 +0200 | [diff] [blame] | 33 | Computes an Adler-32 checksum of *data*. (An Adler-32 checksum is almost as |
Martin Panter | b82032f | 2015-12-11 05:19:29 +0000 | [diff] [blame] | 34 | reliable as a CRC32 but can be computed much more quickly.) The result |
| 35 | is an unsigned 32-bit integer. If *value* is present, it is used as |
| 36 | the starting value of the checksum; otherwise, a default value of 1 |
| 37 | is used. Passing in *value* allows computing a running checksum over the |
Benjamin Peterson | 058e31e | 2009-01-16 03:54:08 +0000 | [diff] [blame] | 38 | concatenation of several inputs. The algorithm is not cryptographically |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 39 | strong, and should not be used for authentication or digital signatures. Since |
| 40 | the algorithm is designed for use as a checksum algorithm, it is not suitable |
| 41 | for use as a general hash algorithm. |
| 42 | |
Martin Panter | b82032f | 2015-12-11 05:19:29 +0000 | [diff] [blame] | 43 | .. versionchanged:: 3.0 |
| 44 | Always returns an unsigned value. |
| 45 | To generate the same numeric value across all Python versions and |
| 46 | platforms, use ``adler32(data) & 0xffffffff``. |
Benjamin Peterson | 058e31e | 2009-01-16 03:54:08 +0000 | [diff] [blame] | 47 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 48 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 49 | .. function:: compress(data[, level]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 50 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 51 | Compresses the bytes in *data*, returning a bytes object containing compressed data. |
Nadeem Vawda | 6ff262e | 2012-11-11 14:14:47 +0100 | [diff] [blame] | 52 | *level* is an integer from ``0`` to ``9`` controlling the level of compression; |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 53 | ``1`` is fastest and produces the least compression, ``9`` is slowest and |
Nadeem Vawda | 6ff262e | 2012-11-11 14:14:47 +0100 | [diff] [blame] | 54 | produces the most. ``0`` is no compression. The default value is ``6``. |
| 55 | Raises the :exc:`error` exception if any error occurs. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 56 | |
| 57 | |
Martin Panter | bf19d16 | 2015-09-09 01:01:13 +0000 | [diff] [blame] | 58 | .. function:: compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 59 | |
| 60 | Returns a compression object, to be used for compressing data streams that won't |
Nadeem Vawda | fd8a838 | 2012-06-21 02:13:12 +0200 | [diff] [blame] | 61 | fit into memory at once. |
| 62 | |
Martin Panter | 567d513 | 2016-02-03 07:06:33 +0000 | [diff] [blame] | 63 | *level* is the compression level -- an integer from ``0`` to ``9`` or ``-1``. |
| 64 | A value of ``1`` is fastest and produces the least compression, while a value of |
Nadeem Vawda | 6ff262e | 2012-11-11 14:14:47 +0100 | [diff] [blame] | 65 | ``9`` is slowest and produces the most. ``0`` is no compression. The default |
Martin Panter | 567d513 | 2016-02-03 07:06:33 +0000 | [diff] [blame] | 66 | value is ``-1`` (Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default |
| 67 | compromise between speed and compression (currently equivalent to level 6). |
Nadeem Vawda | 2180c97 | 2012-06-22 01:40:49 +0200 | [diff] [blame] | 68 | |
| 69 | *method* is the compression algorithm. Currently, the only supported value is |
| 70 | ``DEFLATED``. |
| 71 | |
| 72 | *wbits* is the base two logarithm of the size of the window buffer. This |
| 73 | should be an integer from ``8`` to ``15``. Higher values give better |
| 74 | compression, but use more memory. |
| 75 | |
Martin Panter | bf19d16 | 2015-09-09 01:01:13 +0000 | [diff] [blame] | 76 | The *memLevel* argument controls the amount of memory used for the |
| 77 | internal compression state. Valid values range from ``1`` to ``9``. |
| 78 | Higher values use more memory, but are faster and produce smaller output. |
Nadeem Vawda | 2180c97 | 2012-06-22 01:40:49 +0200 | [diff] [blame] | 79 | |
| 80 | *strategy* is used to tune the compression algorithm. Possible values are |
| 81 | ``Z_DEFAULT_STRATEGY``, ``Z_FILTERED``, and ``Z_HUFFMAN_ONLY``. |
Nadeem Vawda | fd8a838 | 2012-06-21 02:13:12 +0200 | [diff] [blame] | 82 | |
| 83 | *zdict* is a predefined compression dictionary. This is a sequence of bytes |
| 84 | (such as a :class:`bytes` object) containing subsequences that are expected |
| 85 | to occur frequently in the data that is to be compressed. Those subsequences |
| 86 | that are expected to be most common should come at the end of the dictionary. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 87 | |
Georg Brandl | 9aae9e5 | 2012-06-26 08:51:17 +0200 | [diff] [blame] | 88 | .. versionchanged:: 3.3 |
Georg Brandl | 9ff06dc | 2013-10-17 19:51:34 +0200 | [diff] [blame] | 89 | Added the *zdict* parameter and keyword argument support. |
Georg Brandl | 9aae9e5 | 2012-06-26 08:51:17 +0200 | [diff] [blame] | 90 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 91 | |
Benjamin Peterson | 058e31e | 2009-01-16 03:54:08 +0000 | [diff] [blame] | 92 | .. function:: crc32(data[, value]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 93 | |
| 94 | .. index:: |
| 95 | single: Cyclic Redundancy Check |
| 96 | single: checksum; Cyclic Redundancy Check |
| 97 | |
Martin Panter | b82032f | 2015-12-11 05:19:29 +0000 | [diff] [blame] | 98 | Computes a CRC (Cyclic Redundancy Check) checksum of *data*. The |
| 99 | result is an unsigned 32-bit integer. If *value* is present, it is used |
| 100 | as the starting value of the checksum; otherwise, a default value of 0 |
| 101 | is used. Passing in *value* allows computing a running checksum over the |
Benjamin Peterson | 058e31e | 2009-01-16 03:54:08 +0000 | [diff] [blame] | 102 | concatenation of several inputs. The algorithm is not cryptographically |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 103 | strong, and should not be used for authentication or digital signatures. Since |
| 104 | the algorithm is designed for use as a checksum algorithm, it is not suitable |
| 105 | for use as a general hash algorithm. |
| 106 | |
Martin Panter | b82032f | 2015-12-11 05:19:29 +0000 | [diff] [blame] | 107 | .. versionchanged:: 3.0 |
| 108 | Always returns an unsigned value. |
Georg Brandl | 9aae9e5 | 2012-06-26 08:51:17 +0200 | [diff] [blame] | 109 | To generate the same numeric value across all Python versions and |
Martin Panter | b82032f | 2015-12-11 05:19:29 +0000 | [diff] [blame] | 110 | platforms, use ``crc32(data) & 0xffffffff``. |
Benjamin Peterson | 058e31e | 2009-01-16 03:54:08 +0000 | [diff] [blame] | 111 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 112 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 113 | .. function:: decompress(data[, wbits[, bufsize]]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 114 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 115 | Decompresses the bytes in *data*, returning a bytes object containing the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 116 | uncompressed data. The *wbits* parameter controls the size of the window |
Benjamin Peterson | 2614cda | 2010-03-21 22:36:19 +0000 | [diff] [blame] | 117 | buffer, and is discussed further below. |
| 118 | If *bufsize* is given, it is used as the initial size of the output |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 119 | buffer. Raises the :exc:`error` exception if any error occurs. |
| 120 | |
| 121 | The absolute value of *wbits* is the base two logarithm of the size of the |
| 122 | history buffer (the "window size") used when compressing data. Its absolute |
| 123 | value should be between 8 and 15 for the most recent versions of the zlib |
| 124 | library, larger values resulting in better compression at the expense of greater |
Benjamin Peterson | 2614cda | 2010-03-21 22:36:19 +0000 | [diff] [blame] | 125 | memory usage. When decompressing a stream, *wbits* must not be smaller |
| 126 | than the size originally used to compress the stream; using a too-small |
| 127 | value will result in an exception. The default value is therefore the |
| 128 | highest value, 15. When *wbits* is negative, the standard |
Jesus Cea | fb7b668 | 2010-05-03 16:14:58 +0000 | [diff] [blame] | 129 | :program:`gzip` header is suppressed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 130 | |
| 131 | *bufsize* is the initial size of the buffer used to hold decompressed data. If |
| 132 | more space is required, the buffer size will be increased as needed, so you |
| 133 | don't have to get this value exactly right; tuning it will only save a few calls |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 134 | to :c:func:`malloc`. The default size is 16384. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 135 | |
| 136 | |
Georg Brandl | 9aae9e5 | 2012-06-26 08:51:17 +0200 | [diff] [blame] | 137 | .. function:: decompressobj(wbits=15[, zdict]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 138 | |
| 139 | Returns a decompression object, to be used for decompressing data streams that |
Nadeem Vawda | fd8a838 | 2012-06-21 02:13:12 +0200 | [diff] [blame] | 140 | won't fit into memory at once. |
| 141 | |
| 142 | The *wbits* parameter controls the size of the window buffer. |
| 143 | |
| 144 | The *zdict* parameter specifies a predefined compression dictionary. If |
| 145 | provided, this must be the same dictionary as was used by the compressor that |
| 146 | produced the data that is to be decompressed. |
| 147 | |
Georg Brandl | 9aae9e5 | 2012-06-26 08:51:17 +0200 | [diff] [blame] | 148 | .. note:: |
| 149 | |
| 150 | If *zdict* is a mutable object (such as a :class:`bytearray`), you must not |
| 151 | modify its contents between the call to :func:`decompressobj` and the first |
| 152 | call to the decompressor's ``decompress()`` method. |
| 153 | |
| 154 | .. versionchanged:: 3.3 |
| 155 | Added the *zdict* parameter. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 156 | |
Nadeem Vawda | 64d25dd | 2011-09-12 00:04:13 +0200 | [diff] [blame] | 157 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 158 | Compression objects support the following methods: |
| 159 | |
| 160 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 161 | .. method:: Compress.compress(data) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 162 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 163 | Compress *data*, returning a bytes object containing compressed data for at least |
| 164 | part of the data in *data*. This data should be concatenated to the output |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 165 | produced by any preceding calls to the :meth:`compress` method. Some input may |
| 166 | be kept in internal buffers for later processing. |
| 167 | |
| 168 | |
| 169 | .. method:: Compress.flush([mode]) |
| 170 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 171 | All pending input is processed, and a bytes object containing the remaining compressed |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 172 | output is returned. *mode* can be selected from the constants |
| 173 | :const:`Z_SYNC_FLUSH`, :const:`Z_FULL_FLUSH`, or :const:`Z_FINISH`, |
| 174 | defaulting to :const:`Z_FINISH`. :const:`Z_SYNC_FLUSH` and |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 175 | :const:`Z_FULL_FLUSH` allow compressing further bytestrings of data, while |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 176 | :const:`Z_FINISH` finishes the compressed stream and prevents compressing any |
| 177 | more data. After calling :meth:`flush` with *mode* set to :const:`Z_FINISH`, |
| 178 | the :meth:`compress` method cannot be called again; the only realistic action is |
| 179 | to delete the object. |
| 180 | |
| 181 | |
| 182 | .. method:: Compress.copy() |
| 183 | |
| 184 | Returns a copy of the compression object. This can be used to efficiently |
| 185 | compress a set of data that share a common initial prefix. |
| 186 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 187 | |
Nadeem Vawda | 1c38546 | 2011-08-13 15:22:40 +0200 | [diff] [blame] | 188 | Decompression objects support the following methods and attributes: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 189 | |
| 190 | |
| 191 | .. attribute:: Decompress.unused_data |
| 192 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 193 | A bytes object which contains any bytes past the end of the compressed data. That is, |
Serhiy Storchaka | 5e028ae | 2014-02-06 21:10:41 +0200 | [diff] [blame] | 194 | this remains ``b""`` until the last byte that contains compression data is |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 195 | available. If the whole bytestring turned out to contain compressed data, this is |
| 196 | ``b""``, an empty bytes object. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 197 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 198 | |
| 199 | .. attribute:: Decompress.unconsumed_tail |
| 200 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 201 | A bytes object that contains any data that was not consumed by the last |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 202 | :meth:`decompress` call because it exceeded the limit for the uncompressed data |
| 203 | buffer. This data has not yet been seen by the zlib machinery, so you must feed |
| 204 | it (possibly with further data concatenated to it) back to a subsequent |
| 205 | :meth:`decompress` method call in order to get correct output. |
| 206 | |
| 207 | |
Nadeem Vawda | 1c38546 | 2011-08-13 15:22:40 +0200 | [diff] [blame] | 208 | .. attribute:: Decompress.eof |
| 209 | |
| 210 | A boolean indicating whether the end of the compressed data stream has been |
| 211 | reached. |
| 212 | |
| 213 | This makes it possible to distinguish between a properly-formed compressed |
| 214 | stream, and an incomplete or truncated one. |
| 215 | |
| 216 | .. versionadded:: 3.3 |
| 217 | |
| 218 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 219 | .. method:: Decompress.decompress(data[, max_length]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 220 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 221 | Decompress *data*, returning a bytes object containing the uncompressed data |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 222 | corresponding to at least part of the data in *string*. This data should be |
| 223 | concatenated to the output produced by any preceding calls to the |
| 224 | :meth:`decompress` method. Some of the input data may be preserved in internal |
| 225 | buffers for later processing. |
| 226 | |
Martin Panter | 38fe4dc | 2015-11-18 00:59:17 +0000 | [diff] [blame] | 227 | If the optional parameter *max_length* is non-zero then the return value will be |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 228 | no longer than *max_length*. This may mean that not all of the compressed input |
| 229 | can be processed; and unconsumed data will be stored in the attribute |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 230 | :attr:`unconsumed_tail`. This bytestring must be passed to a subsequent call to |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 231 | :meth:`decompress` if decompression is to continue. If *max_length* is not |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 232 | supplied then the whole input is decompressed, and :attr:`unconsumed_tail` is |
| 233 | empty. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 234 | |
| 235 | |
| 236 | .. method:: Decompress.flush([length]) |
| 237 | |
Georg Brandl | 4ad934f | 2011-01-08 21:04:25 +0000 | [diff] [blame] | 238 | All pending input is processed, and a bytes object containing the remaining |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 239 | uncompressed output is returned. After calling :meth:`flush`, the |
| 240 | :meth:`decompress` method cannot be called again; the only realistic action is |
| 241 | to delete the object. |
| 242 | |
| 243 | The optional parameter *length* sets the initial size of the output buffer. |
| 244 | |
| 245 | |
| 246 | .. method:: Decompress.copy() |
| 247 | |
| 248 | Returns a copy of the decompression object. This can be used to save the state |
| 249 | of the decompressor midway through the data stream in order to speed up random |
| 250 | seeks into the stream at a future point. |
| 251 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 252 | |
Nadeem Vawda | 64d25dd | 2011-09-12 00:04:13 +0200 | [diff] [blame] | 253 | Information about the version of the zlib library in use is available through |
| 254 | the following constants: |
| 255 | |
| 256 | |
| 257 | .. data:: ZLIB_VERSION |
| 258 | |
| 259 | The version string of the zlib library that was used for building the module. |
| 260 | This may be different from the zlib library actually used at runtime, which |
| 261 | is available as :const:`ZLIB_RUNTIME_VERSION`. |
| 262 | |
Nadeem Vawda | 64d25dd | 2011-09-12 00:04:13 +0200 | [diff] [blame] | 263 | |
| 264 | .. data:: ZLIB_RUNTIME_VERSION |
| 265 | |
| 266 | The version string of the zlib library actually loaded by the interpreter. |
| 267 | |
| 268 | .. versionadded:: 3.3 |
| 269 | |
| 270 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 271 | .. seealso:: |
| 272 | |
| 273 | Module :mod:`gzip` |
| 274 | Reading and writing :program:`gzip`\ -format files. |
| 275 | |
| 276 | http://www.zlib.net |
| 277 | The zlib library home page. |
| 278 | |
| 279 | http://www.zlib.net/manual.html |
| 280 | The zlib manual explains the semantics and usage of the library's many |
| 281 | functions. |
| 282 | |