The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1 | <?xml version="1.0"?> <!-- -*- sgml -*- --> |
| 2 | <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| 3 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[ |
| 4 | |
| 5 | <!-- various strings, dates etc. common to all docs --> |
| 6 | <!ENTITY % common-ents SYSTEM "entities.xml"> %common-ents; |
| 7 | ]> |
| 8 | |
| 9 | <book lang="en" id="userman" xreflabel="bzip2 Manual"> |
| 10 | |
| 11 | <bookinfo> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 12 | <title>bzip2 and libbzip2, version 1.0.6</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 13 | <subtitle>A program and library for data compression</subtitle> |
| 14 | <copyright> |
| 15 | <year>&bz-lifespan;</year> |
| 16 | <holder>Julian Seward</holder> |
| 17 | </copyright> |
| 18 | <releaseinfo>Version &bz-version; of &bz-date;</releaseinfo> |
| 19 | |
| 20 | <authorgroup> |
| 21 | <author> |
| 22 | <firstname>Julian</firstname> |
| 23 | <surname>Seward</surname> |
| 24 | <affiliation> |
| 25 | <orgname>&bz-url;</orgname> |
| 26 | </affiliation> |
| 27 | </author> |
| 28 | </authorgroup> |
| 29 | |
| 30 | <legalnotice> |
| 31 | |
| 32 | <para>This program, <computeroutput>bzip2</computeroutput>, the |
| 33 | associated library <computeroutput>libbzip2</computeroutput>, and |
| 34 | all documentation, are copyright © &bz-lifespan; Julian Seward. |
| 35 | All rights reserved.</para> |
| 36 | |
| 37 | <para>Redistribution and use in source and binary forms, with |
| 38 | or without modification, are permitted provided that the |
| 39 | following conditions are met:</para> |
| 40 | |
| 41 | <itemizedlist mark='bullet'> |
| 42 | |
| 43 | <listitem><para>Redistributions of source code must retain the |
| 44 | above copyright notice, this list of conditions and the |
| 45 | following disclaimer.</para></listitem> |
| 46 | |
| 47 | <listitem><para>The origin of this software must not be |
| 48 | misrepresented; you must not claim that you wrote the original |
| 49 | software. If you use this software in a product, an |
| 50 | acknowledgment in the product documentation would be |
| 51 | appreciated but is not required.</para></listitem> |
| 52 | |
| 53 | <listitem><para>Altered source versions must be plainly marked |
| 54 | as such, and must not be misrepresented as being the original |
| 55 | software.</para></listitem> |
| 56 | |
| 57 | <listitem><para>The name of the author may not be used to |
| 58 | endorse or promote products derived from this software without |
| 59 | specific prior written permission.</para></listitem> |
| 60 | |
| 61 | </itemizedlist> |
| 62 | |
| 63 | <para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY |
| 64 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, |
| 65 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A |
| 66 | PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE |
| 67 | AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| 68 | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED |
| 69 | TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
| 70 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND |
| 71 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT |
| 72 | LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING |
| 73 | IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF |
| 74 | THE POSSIBILITY OF SUCH DAMAGE.</para> |
| 75 | |
| 76 | <para>PATENTS: To the best of my knowledge, |
| 77 | <computeroutput>bzip2</computeroutput> and |
| 78 | <computeroutput>libbzip2</computeroutput> do not use any patented |
| 79 | algorithms. However, I do not have the resources to carry |
| 80 | out a patent search. Therefore I cannot give any guarantee of |
| 81 | the above statement. |
| 82 | </para> |
| 83 | |
| 84 | </legalnotice> |
| 85 | |
| 86 | </bookinfo> |
| 87 | |
| 88 | |
| 89 | |
| 90 | <chapter id="intro" xreflabel="Introduction"> |
| 91 | <title>Introduction</title> |
| 92 | |
| 93 | <para><computeroutput>bzip2</computeroutput> compresses files |
| 94 | using the Burrows-Wheeler block-sorting text compression |
| 95 | algorithm, and Huffman coding. Compression is generally |
| 96 | considerably better than that achieved by more conventional |
| 97 | LZ77/LZ78-based compressors, and approaches the performance of |
| 98 | the PPM family of statistical compressors.</para> |
| 99 | |
| 100 | <para><computeroutput>bzip2</computeroutput> is built on top of |
| 101 | <computeroutput>libbzip2</computeroutput>, a flexible library for |
| 102 | handling compressed data in the |
| 103 | <computeroutput>bzip2</computeroutput> format. This manual |
| 104 | describes both how to use the program and how to work with the |
| 105 | library interface. Most of the manual is devoted to this |
| 106 | library, not the program, which is good news if your interest is |
| 107 | only in the program.</para> |
| 108 | |
| 109 | <itemizedlist mark='bullet'> |
| 110 | |
| 111 | <listitem><para><xref linkend="using"/> describes how to use |
| 112 | <computeroutput>bzip2</computeroutput>; this is the only part |
| 113 | you need to read if you just want to know how to operate the |
| 114 | program.</para></listitem> |
| 115 | |
| 116 | <listitem><para><xref linkend="libprog"/> describes the |
| 117 | programming interfaces in detail, and</para></listitem> |
| 118 | |
| 119 | <listitem><para><xref linkend="misc"/> records some |
| 120 | miscellaneous notes which I thought ought to be recorded |
| 121 | somewhere.</para></listitem> |
| 122 | |
| 123 | </itemizedlist> |
| 124 | |
| 125 | </chapter> |
| 126 | |
| 127 | |
| 128 | <chapter id="using" xreflabel="How to use bzip2"> |
| 129 | <title>How to use bzip2</title> |
| 130 | |
| 131 | <para>This chapter contains a copy of the |
| 132 | <computeroutput>bzip2</computeroutput> man page, and nothing |
| 133 | else.</para> |
| 134 | |
| 135 | <sect1 id="name" xreflabel="NAME"> |
| 136 | <title>NAME</title> |
| 137 | |
| 138 | <itemizedlist mark='bullet'> |
| 139 | |
| 140 | <listitem><para><computeroutput>bzip2</computeroutput>, |
| 141 | <computeroutput>bunzip2</computeroutput> - a block-sorting file |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 142 | compressor, v1.0.6</para></listitem> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 143 | |
| 144 | <listitem><para><computeroutput>bzcat</computeroutput> - |
| 145 | decompresses files to stdout</para></listitem> |
| 146 | |
| 147 | <listitem><para><computeroutput>bzip2recover</computeroutput> - |
| 148 | recovers data from damaged bzip2 files</para></listitem> |
| 149 | |
| 150 | </itemizedlist> |
| 151 | |
| 152 | </sect1> |
| 153 | |
| 154 | |
| 155 | <sect1 id="synopsis" xreflabel="SYNOPSIS"> |
| 156 | <title>SYNOPSIS</title> |
| 157 | |
| 158 | <itemizedlist mark='bullet'> |
| 159 | |
| 160 | <listitem><para><computeroutput>bzip2</computeroutput> [ |
| 161 | -cdfkqstvzVL123456789 ] [ filenames ... ]</para></listitem> |
| 162 | |
| 163 | <listitem><para><computeroutput>bunzip2</computeroutput> [ |
| 164 | -fkvsVL ] [ filenames ... ]</para></listitem> |
| 165 | |
| 166 | <listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [ |
| 167 | filenames ... ]</para></listitem> |
| 168 | |
| 169 | <listitem><para><computeroutput>bzip2recover</computeroutput> |
| 170 | filename</para></listitem> |
| 171 | |
| 172 | </itemizedlist> |
| 173 | |
| 174 | </sect1> |
| 175 | |
| 176 | |
| 177 | <sect1 id="description" xreflabel="DESCRIPTION"> |
| 178 | <title>DESCRIPTION</title> |
| 179 | |
| 180 | <para><computeroutput>bzip2</computeroutput> compresses files |
| 181 | using the Burrows-Wheeler block sorting text compression |
| 182 | algorithm, and Huffman coding. Compression is generally |
| 183 | considerably better than that achieved by more conventional |
| 184 | LZ77/LZ78-based compressors, and approaches the performance of |
| 185 | the PPM family of statistical compressors.</para> |
| 186 | |
| 187 | <para>The command-line options are deliberately very similar to |
| 188 | those of GNU <computeroutput>gzip</computeroutput>, but they are |
| 189 | not identical.</para> |
| 190 | |
| 191 | <para><computeroutput>bzip2</computeroutput> expects a list of |
| 192 | file names to accompany the command-line flags. Each file is |
| 193 | replaced by a compressed version of itself, with the name |
| 194 | <computeroutput>original_name.bz2</computeroutput>. Each |
| 195 | compressed file has the same modification date, permissions, and, |
| 196 | when possible, ownership as the corresponding original, so that |
| 197 | these properties can be correctly restored at decompression time. |
| 198 | File name handling is naive in the sense that there is no |
| 199 | mechanism for preserving original file names, permissions, |
| 200 | ownerships or dates in filesystems which lack these concepts, or |
| 201 | have serious file name length restrictions, such as |
| 202 | MS-DOS.</para> |
| 203 | |
| 204 | <para><computeroutput>bzip2</computeroutput> and |
| 205 | <computeroutput>bunzip2</computeroutput> will by default not |
| 206 | overwrite existing files. If you want this to happen, specify |
| 207 | the <computeroutput>-f</computeroutput> flag.</para> |
| 208 | |
| 209 | <para>If no file names are specified, |
| 210 | <computeroutput>bzip2</computeroutput> compresses from standard |
| 211 | input to standard output. In this case, |
| 212 | <computeroutput>bzip2</computeroutput> will decline to write |
| 213 | compressed output to a terminal, as this would be entirely |
| 214 | incomprehensible and therefore pointless.</para> |
| 215 | |
| 216 | <para><computeroutput>bunzip2</computeroutput> (or |
| 217 | <computeroutput>bzip2 -d</computeroutput>) decompresses all |
| 218 | specified files. Files which were not created by |
| 219 | <computeroutput>bzip2</computeroutput> will be detected and |
| 220 | ignored, and a warning issued. |
| 221 | <computeroutput>bzip2</computeroutput> attempts to guess the |
| 222 | filename for the decompressed file from that of the compressed |
| 223 | file as follows:</para> |
| 224 | |
| 225 | <itemizedlist mark='bullet'> |
| 226 | |
| 227 | <listitem><para><computeroutput>filename.bz2 </computeroutput> |
| 228 | becomes |
| 229 | <computeroutput>filename</computeroutput></para></listitem> |
| 230 | |
| 231 | <listitem><para><computeroutput>filename.bz </computeroutput> |
| 232 | becomes |
| 233 | <computeroutput>filename</computeroutput></para></listitem> |
| 234 | |
| 235 | <listitem><para><computeroutput>filename.tbz2</computeroutput> |
| 236 | becomes |
| 237 | <computeroutput>filename.tar</computeroutput></para></listitem> |
| 238 | |
| 239 | <listitem><para><computeroutput>filename.tbz </computeroutput> |
| 240 | becomes |
| 241 | <computeroutput>filename.tar</computeroutput></para></listitem> |
| 242 | |
| 243 | <listitem><para><computeroutput>anyothername </computeroutput> |
| 244 | becomes |
| 245 | <computeroutput>anyothername.out</computeroutput></para></listitem> |
| 246 | |
| 247 | </itemizedlist> |
| 248 | |
| 249 | <para>If the file does not end in one of the recognised endings, |
| 250 | <computeroutput>.bz2</computeroutput>, |
| 251 | <computeroutput>.bz</computeroutput>, |
| 252 | <computeroutput>.tbz2</computeroutput> or |
| 253 | <computeroutput>.tbz</computeroutput>, |
| 254 | <computeroutput>bzip2</computeroutput> complains that it cannot |
| 255 | guess the name of the original file, and uses the original name |
| 256 | with <computeroutput>.out</computeroutput> appended.</para> |
| 257 | |
| 258 | <para>As with compression, supplying no filenames causes |
| 259 | decompression from standard input to standard output.</para> |
| 260 | |
| 261 | <para><computeroutput>bunzip2</computeroutput> will correctly |
| 262 | decompress a file which is the concatenation of two or more |
| 263 | compressed files. The result is the concatenation of the |
| 264 | corresponding uncompressed files. Integrity testing |
| 265 | (<computeroutput>-t</computeroutput>) of concatenated compressed |
| 266 | files is also supported.</para> |
| 267 | |
| 268 | <para>You can also compress or decompress files to the standard |
| 269 | output by giving the <computeroutput>-c</computeroutput> flag. |
| 270 | Multiple files may be compressed and decompressed like this. The |
| 271 | resulting outputs are fed sequentially to stdout. Compression of |
| 272 | multiple files in this manner generates a stream containing |
| 273 | multiple compressed file representations. Such a stream can be |
| 274 | decompressed correctly only by |
| 275 | <computeroutput>bzip2</computeroutput> version 0.9.0 or later. |
| 276 | Earlier versions of <computeroutput>bzip2</computeroutput> will |
| 277 | stop after decompressing the first file in the stream.</para> |
| 278 | |
| 279 | <para><computeroutput>bzcat</computeroutput> (or |
| 280 | <computeroutput>bzip2 -dc</computeroutput>) decompresses all |
| 281 | specified files to the standard output.</para> |
| 282 | |
| 283 | <para><computeroutput>bzip2</computeroutput> will read arguments |
| 284 | from the environment variables |
| 285 | <computeroutput>BZIP2</computeroutput> and |
| 286 | <computeroutput>BZIP</computeroutput>, in that order, and will |
| 287 | process them before any arguments read from the command line. |
| 288 | This gives a convenient way to supply default arguments.</para> |
| 289 | |
| 290 | <para>Compression is always performed, even if the compressed |
| 291 | file is slightly larger than the original. Files of less than |
| 292 | about one hundred bytes tend to get larger, since the compression |
| 293 | mechanism has a constant overhead in the region of 50 bytes. |
| 294 | Random data (including the output of most file compressors) is |
| 295 | coded at about 8.05 bits per byte, giving an expansion of around |
| 296 | 0.5%.</para> |
| 297 | |
| 298 | <para>As a self-check for your protection, |
| 299 | <computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make |
| 300 | sure that the decompressed version of a file is identical to the |
| 301 | original. This guards against corruption of the compressed data, |
| 302 | and against undetected bugs in |
| 303 | <computeroutput>bzip2</computeroutput> (hopefully very unlikely). |
| 304 | The chances of data corruption going undetected is microscopic, |
| 305 | about one chance in four billion for each file processed. Be |
| 306 | aware, though, that the check occurs upon decompression, so it |
| 307 | can only tell you that something is wrong. It can't help you |
| 308 | recover the original uncompressed data. You can use |
| 309 | <computeroutput>bzip2recover</computeroutput> to try to recover |
| 310 | data from damaged files.</para> |
| 311 | |
| 312 | <para>Return values: 0 for a normal exit, 1 for environmental |
| 313 | problems (file not found, invalid flags, I/O errors, etc.), 2 |
| 314 | to indicate a corrupt compressed file, 3 for an internal |
| 315 | consistency error (eg, bug) which caused |
| 316 | <computeroutput>bzip2</computeroutput> to panic.</para> |
| 317 | |
| 318 | </sect1> |
| 319 | |
| 320 | |
| 321 | <sect1 id="options" xreflabel="OPTIONS"> |
| 322 | <title>OPTIONS</title> |
| 323 | |
| 324 | <variablelist> |
| 325 | |
| 326 | <varlistentry> |
| 327 | <term><computeroutput>-c --stdout</computeroutput></term> |
| 328 | <listitem><para>Compress or decompress to standard |
| 329 | output.</para></listitem> |
| 330 | </varlistentry> |
| 331 | |
| 332 | <varlistentry> |
| 333 | <term><computeroutput>-d --decompress</computeroutput></term> |
| 334 | <listitem><para>Force decompression. |
| 335 | <computeroutput>bzip2</computeroutput>, |
| 336 | <computeroutput>bunzip2</computeroutput> and |
| 337 | <computeroutput>bzcat</computeroutput> are really the same |
| 338 | program, and the decision about what actions to take is done on |
| 339 | the basis of which name is used. This flag overrides that |
| 340 | mechanism, and forces bzip2 to decompress.</para></listitem> |
| 341 | </varlistentry> |
| 342 | |
| 343 | <varlistentry> |
| 344 | <term><computeroutput>-z --compress</computeroutput></term> |
| 345 | <listitem><para>The complement to |
| 346 | <computeroutput>-d</computeroutput>: forces compression, |
| 347 | regardless of the invokation name.</para></listitem> |
| 348 | </varlistentry> |
| 349 | |
| 350 | <varlistentry> |
| 351 | <term><computeroutput>-t --test</computeroutput></term> |
| 352 | <listitem><para>Check integrity of the specified file(s), but |
| 353 | don't decompress them. This really performs a trial |
| 354 | decompression and throws away the result.</para></listitem> |
| 355 | </varlistentry> |
| 356 | |
| 357 | <varlistentry> |
| 358 | <term><computeroutput>-f --force</computeroutput></term> |
| 359 | <listitem><para>Force overwrite of output files. Normally, |
| 360 | <computeroutput>bzip2</computeroutput> will not overwrite |
| 361 | existing output files. Also forces |
| 362 | <computeroutput>bzip2</computeroutput> to break hard links to |
| 363 | files, which it otherwise wouldn't do.</para> |
| 364 | <para><computeroutput>bzip2</computeroutput> normally declines |
| 365 | to decompress files which don't have the correct magic header |
| 366 | bytes. If forced (<computeroutput>-f</computeroutput>), |
| 367 | however, it will pass such files through unmodified. This is |
| 368 | how GNU <computeroutput>gzip</computeroutput> behaves.</para> |
| 369 | </listitem> |
| 370 | </varlistentry> |
| 371 | |
| 372 | <varlistentry> |
| 373 | <term><computeroutput>-k --keep</computeroutput></term> |
| 374 | <listitem><para>Keep (don't delete) input files during |
| 375 | compression or decompression.</para></listitem> |
| 376 | </varlistentry> |
| 377 | |
| 378 | <varlistentry> |
| 379 | <term><computeroutput>-s --small</computeroutput></term> |
| 380 | <listitem><para>Reduce memory usage, for compression, |
| 381 | decompression and testing. Files are decompressed and tested |
| 382 | using a modified algorithm which only requires 2.5 bytes per |
| 383 | block byte. This means any file can be decompressed in 2300k |
| 384 | of memory, albeit at about half the normal speed.</para> |
| 385 | <para>During compression, <computeroutput>-s</computeroutput> |
| 386 | selects a block size of 200k, which limits memory use to around |
| 387 | the same figure, at the expense of your compression ratio. In |
| 388 | short, if your machine is low on memory (8 megabytes or less), |
| 389 | use <computeroutput>-s</computeroutput> for everything. See |
| 390 | <xref linkend="memory-management"/> below.</para></listitem> |
| 391 | </varlistentry> |
| 392 | |
| 393 | <varlistentry> |
| 394 | <term><computeroutput>-q --quiet</computeroutput></term> |
| 395 | <listitem><para>Suppress non-essential warning messages. |
| 396 | Messages pertaining to I/O errors and other critical events |
| 397 | will not be suppressed.</para></listitem> |
| 398 | </varlistentry> |
| 399 | |
| 400 | <varlistentry> |
| 401 | <term><computeroutput>-v --verbose</computeroutput></term> |
| 402 | <listitem><para>Verbose mode -- show the compression ratio for |
| 403 | each file processed. Further |
| 404 | <computeroutput>-v</computeroutput>'s increase the verbosity |
| 405 | level, spewing out lots of information which is primarily of |
| 406 | interest for diagnostic purposes.</para></listitem> |
| 407 | </varlistentry> |
| 408 | |
| 409 | <varlistentry> |
| 410 | <term><computeroutput>-L --license -V --version</computeroutput></term> |
| 411 | <listitem><para>Display the software version, license terms and |
| 412 | conditions.</para></listitem> |
| 413 | </varlistentry> |
| 414 | |
| 415 | <varlistentry> |
| 416 | <term><computeroutput>-1</computeroutput> (or |
| 417 | <computeroutput>--fast</computeroutput>) to |
| 418 | <computeroutput>-9</computeroutput> (or |
| 419 | <computeroutput>-best</computeroutput>)</term> |
| 420 | <listitem><para>Set the block size to 100 k, 200 k ... 900 k |
| 421 | when compressing. Has no effect when decompressing. See <xref |
| 422 | linkend="memory-management" /> below. The |
| 423 | <computeroutput>--fast</computeroutput> and |
| 424 | <computeroutput>--best</computeroutput> aliases are primarily |
| 425 | for GNU <computeroutput>gzip</computeroutput> compatibility. |
| 426 | In particular, <computeroutput>--fast</computeroutput> doesn't |
| 427 | make things significantly faster. And |
| 428 | <computeroutput>--best</computeroutput> merely selects the |
| 429 | default behaviour.</para></listitem> |
| 430 | </varlistentry> |
| 431 | |
| 432 | <varlistentry> |
| 433 | <term><computeroutput>--</computeroutput></term> |
| 434 | <listitem><para>Treats all subsequent arguments as file names, |
| 435 | even if they start with a dash. This is so you can handle |
| 436 | files with names beginning with a dash, for example: |
| 437 | <computeroutput>bzip2 -- |
| 438 | -myfilename</computeroutput>.</para></listitem> |
| 439 | </varlistentry> |
| 440 | |
| 441 | <varlistentry> |
| 442 | <term><computeroutput>--repetitive-fast</computeroutput></term> |
| 443 | <term><computeroutput>--repetitive-best</computeroutput></term> |
| 444 | <listitem><para>These flags are redundant in versions 0.9.5 and |
| 445 | above. They provided some coarse control over the behaviour of |
| 446 | the sorting algorithm in earlier versions, which was sometimes |
| 447 | useful. 0.9.5 and above have an improved algorithm which |
| 448 | renders these flags irrelevant.</para></listitem> |
| 449 | </varlistentry> |
| 450 | |
| 451 | </variablelist> |
| 452 | |
| 453 | </sect1> |
| 454 | |
| 455 | |
| 456 | <sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT"> |
| 457 | <title>MEMORY MANAGEMENT</title> |
| 458 | |
| 459 | <para><computeroutput>bzip2</computeroutput> compresses large |
| 460 | files in blocks. The block size affects both the compression |
| 461 | ratio achieved, and the amount of memory needed for compression |
| 462 | and decompression. The flags <computeroutput>-1</computeroutput> |
| 463 | through <computeroutput>-9</computeroutput> specify the block |
| 464 | size to be 100,000 bytes through 900,000 bytes (the default) |
| 465 | respectively. At decompression time, the block size used for |
| 466 | compression is read from the header of the compressed file, and |
| 467 | <computeroutput>bunzip2</computeroutput> then allocates itself |
| 468 | just enough memory to decompress the file. Since block sizes are |
| 469 | stored in compressed files, it follows that the flags |
| 470 | <computeroutput>-1</computeroutput> to |
| 471 | <computeroutput>-9</computeroutput> are irrelevant to and so |
| 472 | ignored during decompression.</para> |
| 473 | |
| 474 | <para>Compression and decompression requirements, in bytes, can be |
| 475 | estimated as:</para> |
| 476 | <programlisting> |
| 477 | Compression: 400k + ( 8 x block size ) |
| 478 | |
| 479 | Decompression: 100k + ( 4 x block size ), or |
| 480 | 100k + ( 2.5 x block size ) |
| 481 | </programlisting> |
| 482 | |
| 483 | <para>Larger block sizes give rapidly diminishing marginal |
| 484 | returns. Most of the compression comes from the first two or |
| 485 | three hundred k of block size, a fact worth bearing in mind when |
| 486 | using <computeroutput>bzip2</computeroutput> on small machines. |
| 487 | It is also important to appreciate that the decompression memory |
| 488 | requirement is set at compression time by the choice of block |
| 489 | size.</para> |
| 490 | |
| 491 | <para>For files compressed with the default 900k block size, |
| 492 | <computeroutput>bunzip2</computeroutput> will require about 3700 |
| 493 | kbytes to decompress. To support decompression of any file on a |
| 494 | 4 megabyte machine, <computeroutput>bunzip2</computeroutput> has |
| 495 | an option to decompress using approximately half this amount of |
| 496 | memory, about 2300 kbytes. Decompression speed is also halved, |
| 497 | so you should use this option only where necessary. The relevant |
| 498 | flag is <computeroutput>-s</computeroutput>.</para> |
| 499 | |
| 500 | <para>In general, try and use the largest block size memory |
| 501 | constraints allow, since that maximises the compression achieved. |
| 502 | Compression and decompression speed are virtually unaffected by |
| 503 | block size.</para> |
| 504 | |
| 505 | <para>Another significant point applies to files which fit in a |
| 506 | single block -- that means most files you'd encounter using a |
| 507 | large block size. The amount of real memory touched is |
| 508 | proportional to the size of the file, since the file is smaller |
| 509 | than a block. For example, compressing a file 20,000 bytes long |
| 510 | with the flag <computeroutput>-9</computeroutput> will cause the |
| 511 | compressor to allocate around 7600k of memory, but only touch |
| 512 | 400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor |
| 513 | will allocate 3700k but only touch 100k + 20000 * 4 = 180 |
| 514 | kbytes.</para> |
| 515 | |
| 516 | <para>Here is a table which summarises the maximum memory usage |
| 517 | for different block sizes. Also recorded is the total compressed |
| 518 | size for 14 files of the Calgary Text Compression Corpus |
| 519 | totalling 3,141,622 bytes. This column gives some feel for how |
| 520 | compression varies with block size. These figures tend to |
| 521 | understate the advantage of larger block sizes for larger files, |
| 522 | since the Corpus is dominated by smaller files.</para> |
| 523 | |
| 524 | <programlisting> |
| 525 | Compress Decompress Decompress Corpus |
| 526 | Flag usage usage -s usage Size |
| 527 | |
| 528 | -1 1200k 500k 350k 914704 |
| 529 | -2 2000k 900k 600k 877703 |
| 530 | -3 2800k 1300k 850k 860338 |
| 531 | -4 3600k 1700k 1100k 846899 |
| 532 | -5 4400k 2100k 1350k 845160 |
| 533 | -6 5200k 2500k 1600k 838626 |
| 534 | -7 6100k 2900k 1850k 834096 |
| 535 | -8 6800k 3300k 2100k 828642 |
| 536 | -9 7600k 3700k 2350k 828642 |
| 537 | </programlisting> |
| 538 | |
| 539 | </sect1> |
| 540 | |
| 541 | |
| 542 | <sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES"> |
| 543 | <title>RECOVERING DATA FROM DAMAGED FILES</title> |
| 544 | |
| 545 | <para><computeroutput>bzip2</computeroutput> compresses files in |
| 546 | blocks, usually 900kbytes long. Each block is handled |
| 547 | independently. If a media or transmission error causes a |
| 548 | multi-block <computeroutput>.bz2</computeroutput> file to become |
| 549 | damaged, it may be possible to recover data from the undamaged |
| 550 | blocks in the file.</para> |
| 551 | |
| 552 | <para>The compressed representation of each block is delimited by |
| 553 | a 48-bit pattern, which makes it possible to find the block |
| 554 | boundaries with reasonable certainty. Each block also carries |
| 555 | its own 32-bit CRC, so damaged blocks can be distinguished from |
| 556 | undamaged ones.</para> |
| 557 | |
| 558 | <para><computeroutput>bzip2recover</computeroutput> is a simple |
| 559 | program whose purpose is to search for blocks in |
| 560 | <computeroutput>.bz2</computeroutput> files, and write each block |
| 561 | out into its own <computeroutput>.bz2</computeroutput> file. You |
| 562 | can then use <computeroutput>bzip2 -t</computeroutput> to test |
| 563 | the integrity of the resulting files, and decompress those which |
| 564 | are undamaged.</para> |
| 565 | |
| 566 | <para><computeroutput>bzip2recover</computeroutput> takes a |
| 567 | single argument, the name of the damaged file, and writes a |
| 568 | number of files <computeroutput>rec0001file.bz2</computeroutput>, |
| 569 | <computeroutput>rec0002file.bz2</computeroutput>, etc, containing |
| 570 | the extracted blocks. The output filenames are designed so that |
| 571 | the use of wildcards in subsequent processing -- for example, |
| 572 | <computeroutput>bzip2 -dc rec*file.bz2 > |
| 573 | recovered_data</computeroutput> -- lists the files in the correct |
| 574 | order.</para> |
| 575 | |
| 576 | <para><computeroutput>bzip2recover</computeroutput> should be of |
| 577 | most use dealing with large <computeroutput>.bz2</computeroutput> |
| 578 | files, as these will contain many blocks. It is clearly futile |
| 579 | to use it on damaged single-block files, since a damaged block |
| 580 | cannot be recovered. If you wish to minimise any potential data |
| 581 | loss through media or transmission errors, you might consider |
| 582 | compressing with a smaller block size.</para> |
| 583 | |
| 584 | </sect1> |
| 585 | |
| 586 | |
| 587 | <sect1 id="performance" xreflabel="PERFORMANCE NOTES"> |
| 588 | <title>PERFORMANCE NOTES</title> |
| 589 | |
| 590 | <para>The sorting phase of compression gathers together similar |
| 591 | strings in the file. Because of this, files containing very long |
| 592 | runs of repeated symbols, like "aabaabaabaab ..." (repeated |
| 593 | several hundred times) may compress more slowly than normal. |
| 594 | Versions 0.9.5 and above fare much better than previous versions |
| 595 | in this respect. The ratio between worst-case and average-case |
| 596 | compression time is in the region of 10:1. For previous |
| 597 | versions, this figure was more like 100:1. You can use the |
| 598 | <computeroutput>-vvvv</computeroutput> option to monitor progress |
| 599 | in great detail, if you want.</para> |
| 600 | |
| 601 | <para>Decompression speed is unaffected by these |
| 602 | phenomena.</para> |
| 603 | |
| 604 | <para><computeroutput>bzip2</computeroutput> usually allocates |
| 605 | several megabytes of memory to operate in, and then charges all |
| 606 | over it in a fairly random fashion. This means that performance, |
| 607 | both for compressing and decompressing, is largely determined by |
| 608 | the speed at which your machine can service cache misses. |
| 609 | Because of this, small changes to the code to reduce the miss |
| 610 | rate have been observed to give disproportionately large |
| 611 | performance improvements. I imagine |
| 612 | <computeroutput>bzip2</computeroutput> will perform best on |
| 613 | machines with very large caches.</para> |
| 614 | |
| 615 | </sect1> |
| 616 | |
| 617 | |
| 618 | |
| 619 | <sect1 id="caveats" xreflabel="CAVEATS"> |
| 620 | <title>CAVEATS</title> |
| 621 | |
| 622 | <para>I/O error messages are not as helpful as they could be. |
| 623 | <computeroutput>bzip2</computeroutput> tries hard to detect I/O |
| 624 | errors and exit cleanly, but the details of what the problem is |
| 625 | sometimes seem rather misleading.</para> |
| 626 | |
| 627 | <para>This manual page pertains to version &bz-version; of |
| 628 | <computeroutput>bzip2</computeroutput>. Compressed data created by |
| 629 | this version is entirely forwards and backwards compatible with the |
| 630 | previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0, |
| 631 | 1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and |
| 632 | above can correctly decompress multiple concatenated compressed files. |
| 633 | 0.1pl2 cannot do this; it will stop after decompressing just the first |
| 634 | file in the stream.</para> |
| 635 | |
| 636 | <para><computeroutput>bzip2recover</computeroutput> versions |
| 637 | prior to 1.0.2 used 32-bit integers to represent bit positions in |
| 638 | compressed files, so it could not handle compressed files more |
| 639 | than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints |
| 640 | on some platforms which support them (GNU supported targets, and |
| 641 | Windows). To establish whether or not |
| 642 | <computeroutput>bzip2recover</computeroutput> was built with such |
| 643 | a limitation, run it without arguments. In any event you can |
| 644 | build yourself an unlimited version if you can recompile it with |
| 645 | <computeroutput>MaybeUInt64</computeroutput> set to be an |
| 646 | unsigned 64-bit integer.</para> |
| 647 | |
| 648 | </sect1> |
| 649 | |
| 650 | |
| 651 | |
| 652 | <sect1 id="author" xreflabel="AUTHOR"> |
| 653 | <title>AUTHOR</title> |
| 654 | |
| 655 | <para>Julian Seward, |
| 656 | <computeroutput>&bz-email;</computeroutput></para> |
| 657 | |
| 658 | <para>The ideas embodied in |
| 659 | <computeroutput>bzip2</computeroutput> are due to (at least) the |
| 660 | following people: Michael Burrows and David Wheeler (for the |
| 661 | block sorting transformation), David Wheeler (again, for the |
| 662 | Huffman coder), Peter Fenwick (for the structured coding model in |
| 663 | the original <computeroutput>bzip</computeroutput>, and many |
| 664 | refinements), and Alistair Moffat, Radford Neal and Ian Witten |
| 665 | (for the arithmetic coder in the original |
| 666 | <computeroutput>bzip</computeroutput>). I am much indebted for |
| 667 | their help, support and advice. See the manual in the source |
| 668 | distribution for pointers to sources of documentation. Christian |
| 669 | von Roques encouraged me to look for faster sorting algorithms, |
| 670 | so as to speed up compression. Bela Lubkin encouraged me to |
| 671 | improve the worst-case compression performance. |
| 672 | Donna Robinson XMLised the documentation. |
| 673 | Many people sent |
| 674 | patches, helped with portability problems, lent machines, gave |
| 675 | advice and were generally helpful.</para> |
| 676 | |
| 677 | </sect1> |
| 678 | |
| 679 | </chapter> |
| 680 | |
| 681 | |
| 682 | |
| 683 | <chapter id="libprog" xreflabel="Programming with libbzip2"> |
| 684 | <title> |
| 685 | Programming with <computeroutput>libbzip2</computeroutput> |
| 686 | </title> |
| 687 | |
| 688 | <para>This chapter describes the programming interface to |
| 689 | <computeroutput>libbzip2</computeroutput>.</para> |
| 690 | |
| 691 | <para>For general background information, particularly about |
| 692 | memory use and performance aspects, you'd be well advised to read |
| 693 | <xref linkend="using"/> as well.</para> |
| 694 | |
| 695 | |
| 696 | <sect1 id="top-level" xreflabel="Top-level structure"> |
| 697 | <title>Top-level structure</title> |
| 698 | |
| 699 | <para><computeroutput>libbzip2</computeroutput> is a flexible |
| 700 | library for compressing and decompressing data in the |
| 701 | <computeroutput>bzip2</computeroutput> data format. Although |
| 702 | packaged as a single entity, it helps to regard the library as |
| 703 | three separate parts: the low level interface, and the high level |
| 704 | interface, and some utility functions.</para> |
| 705 | |
| 706 | <para>The structure of |
| 707 | <computeroutput>libbzip2</computeroutput>'s interfaces is similar |
| 708 | to that of Jean-loup Gailly's and Mark Adler's excellent |
| 709 | <computeroutput>zlib</computeroutput> library.</para> |
| 710 | |
| 711 | <para>All externally visible symbols have names beginning |
| 712 | <computeroutput>BZ2_</computeroutput>. This is new in version |
| 713 | 1.0. The intention is to minimise pollution of the namespaces of |
| 714 | library clients.</para> |
| 715 | |
| 716 | <para>To use any part of the library, you need to |
| 717 | <computeroutput>#include <bzlib.h></computeroutput> |
| 718 | into your sources.</para> |
| 719 | |
| 720 | |
| 721 | |
| 722 | <sect2 id="ll-summary" xreflabel="Low-level summary"> |
| 723 | <title>Low-level summary</title> |
| 724 | |
| 725 | <para>This interface provides services for compressing and |
| 726 | decompressing data in memory. There's no provision for dealing |
| 727 | with files, streams or any other I/O mechanisms, just straight |
| 728 | memory-to-memory work. In fact, this part of the library can be |
| 729 | compiled without inclusion of |
| 730 | <computeroutput>stdio.h</computeroutput>, which may be helpful |
| 731 | for embedded applications.</para> |
| 732 | |
| 733 | <para>The low-level part of the library has no global variables |
| 734 | and is therefore thread-safe.</para> |
| 735 | |
| 736 | <para>Six routines make up the low level interface: |
| 737 | <computeroutput>BZ2_bzCompressInit</computeroutput>, |
| 738 | <computeroutput>BZ2_bzCompress</computeroutput>, and |
| 739 | <computeroutput>BZ2_bzCompressEnd</computeroutput> for |
| 740 | compression, and a corresponding trio |
| 741 | <computeroutput>BZ2_bzDecompressInit</computeroutput>, |
| 742 | <computeroutput>BZ2_bzDecompress</computeroutput> and |
| 743 | <computeroutput>BZ2_bzDecompressEnd</computeroutput> for |
| 744 | decompression. The <computeroutput>*Init</computeroutput> |
| 745 | functions allocate memory for compression/decompression and do |
| 746 | other initialisations, whilst the |
| 747 | <computeroutput>*End</computeroutput> functions close down |
| 748 | operations and release memory.</para> |
| 749 | |
| 750 | <para>The real work is done by |
| 751 | <computeroutput>BZ2_bzCompress</computeroutput> and |
| 752 | <computeroutput>BZ2_bzDecompress</computeroutput>. These |
| 753 | compress and decompress data from a user-supplied input buffer to |
| 754 | a user-supplied output buffer. These buffers can be any size; |
| 755 | arbitrary quantities of data are handled by making repeated calls |
| 756 | to these functions. This is a flexible mechanism allowing a |
| 757 | consumer-pull style of activity, or producer-push, or a mixture |
| 758 | of both.</para> |
| 759 | |
| 760 | </sect2> |
| 761 | |
| 762 | |
| 763 | <sect2 id="hl-summary" xreflabel="High-level summary"> |
| 764 | <title>High-level summary</title> |
| 765 | |
| 766 | <para>This interface provides some handy wrappers around the |
| 767 | low-level interface to facilitate reading and writing |
| 768 | <computeroutput>bzip2</computeroutput> format files |
| 769 | (<computeroutput>.bz2</computeroutput> files). The routines |
| 770 | provide hooks to facilitate reading files in which the |
| 771 | <computeroutput>bzip2</computeroutput> data stream is embedded |
| 772 | within some larger-scale file structure, or where there are |
| 773 | multiple <computeroutput>bzip2</computeroutput> data streams |
| 774 | concatenated end-to-end.</para> |
| 775 | |
| 776 | <para>For reading files, |
| 777 | <computeroutput>BZ2_bzReadOpen</computeroutput>, |
| 778 | <computeroutput>BZ2_bzRead</computeroutput>, |
| 779 | <computeroutput>BZ2_bzReadClose</computeroutput> and |
| 780 | <computeroutput>BZ2_bzReadGetUnused</computeroutput> are |
| 781 | supplied. For writing files, |
| 782 | <computeroutput>BZ2_bzWriteOpen</computeroutput>, |
| 783 | <computeroutput>BZ2_bzWrite</computeroutput> and |
| 784 | <computeroutput>BZ2_bzWriteFinish</computeroutput> are |
| 785 | available.</para> |
| 786 | |
| 787 | <para>As with the low-level library, no global variables are used |
| 788 | so the library is per se thread-safe. However, if I/O errors |
| 789 | occur whilst reading or writing the underlying compressed files, |
| 790 | you may have to consult <computeroutput>errno</computeroutput> to |
| 791 | determine the cause of the error. In that case, you'd need a C |
| 792 | library which correctly supports |
| 793 | <computeroutput>errno</computeroutput> in a multithreaded |
| 794 | environment.</para> |
| 795 | |
| 796 | <para>To make the library a little simpler and more portable, |
| 797 | <computeroutput>BZ2_bzReadOpen</computeroutput> and |
| 798 | <computeroutput>BZ2_bzWriteOpen</computeroutput> require you to |
| 799 | pass them file handles (<computeroutput>FILE*</computeroutput>s) |
| 800 | which have previously been opened for reading or writing |
| 801 | respectively. That avoids portability problems associated with |
| 802 | file operations and file attributes, whilst not being much of an |
| 803 | imposition on the programmer.</para> |
| 804 | |
| 805 | </sect2> |
| 806 | |
| 807 | |
| 808 | <sect2 id="util-fns-summary" xreflabel="Utility functions summary"> |
| 809 | <title>Utility functions summary</title> |
| 810 | |
| 811 | <para>For very simple needs, |
| 812 | <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and |
| 813 | <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are |
| 814 | provided. These compress data in memory from one buffer to |
| 815 | another buffer in a single function call. You should assess |
| 816 | whether these functions fulfill your memory-to-memory |
| 817 | compression/decompression requirements before investing effort in |
| 818 | understanding the more general but more complex low-level |
| 819 | interface.</para> |
| 820 | |
| 821 | <para>Yoshioka Tsuneo |
| 822 | (<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has |
| 823 | contributed some functions to give better |
| 824 | <computeroutput>zlib</computeroutput> compatibility. These |
| 825 | functions are <computeroutput>BZ2_bzopen</computeroutput>, |
| 826 | <computeroutput>BZ2_bzread</computeroutput>, |
| 827 | <computeroutput>BZ2_bzwrite</computeroutput>, |
| 828 | <computeroutput>BZ2_bzflush</computeroutput>, |
| 829 | <computeroutput>BZ2_bzclose</computeroutput>, |
| 830 | <computeroutput>BZ2_bzerror</computeroutput> and |
| 831 | <computeroutput>BZ2_bzlibVersion</computeroutput>. You may find |
| 832 | these functions more convenient for simple file reading and |
| 833 | writing, than those in the high-level interface. These functions |
| 834 | are not (yet) officially part of the library, and are minimally |
| 835 | documented here. If they break, you get to keep all the pieces. |
| 836 | I hope to document them properly when time permits.</para> |
| 837 | |
| 838 | <para>Yoshioka also contributed modifications to allow the |
| 839 | library to be built as a Windows DLL.</para> |
| 840 | |
| 841 | </sect2> |
| 842 | |
| 843 | </sect1> |
| 844 | |
| 845 | |
| 846 | <sect1 id="err-handling" xreflabel="Error handling"> |
| 847 | <title>Error handling</title> |
| 848 | |
| 849 | <para>The library is designed to recover cleanly in all |
| 850 | situations, including the worst-case situation of decompressing |
| 851 | random data. I'm not 100% sure that it can always do this, so |
| 852 | you might want to add a signal handler to catch segmentation |
| 853 | violations during decompression if you are feeling especially |
| 854 | paranoid. I would be interested in hearing more about the |
| 855 | robustness of the library to corrupted compressed data.</para> |
| 856 | |
| 857 | <para>Version 1.0.3 more robust in this respect than any |
| 858 | previous version. Investigations with Valgrind (a tool for detecting |
| 859 | problems with memory management) indicate |
| 860 | that, at least for the few files I tested, all single-bit errors |
| 861 | in the decompressed data are caught properly, with no |
| 862 | segmentation faults, no uses of uninitialised data, no out of |
| 863 | range reads or writes, and no infinite looping in the decompressor. |
| 864 | So it's certainly pretty robust, although |
| 865 | I wouldn't claim it to be totally bombproof.</para> |
| 866 | |
| 867 | <para>The file <computeroutput>bzlib.h</computeroutput> contains |
| 868 | all definitions needed to use the library. In particular, you |
| 869 | should definitely not include |
| 870 | <computeroutput>bzlib_private.h</computeroutput>.</para> |
| 871 | |
| 872 | <para>In <computeroutput>bzlib.h</computeroutput>, the various |
| 873 | return values are defined. The following list is not intended as |
| 874 | an exhaustive description of the circumstances in which a given |
| 875 | value may be returned -- those descriptions are given later. |
| 876 | Rather, it is intended to convey the rough meaning of each return |
| 877 | value. The first five actions are normal and not intended to |
| 878 | denote an error situation.</para> |
| 879 | |
| 880 | <variablelist> |
| 881 | |
| 882 | <varlistentry> |
| 883 | <term><computeroutput>BZ_OK</computeroutput></term> |
| 884 | <listitem><para>The requested action was completed |
| 885 | successfully.</para></listitem> |
| 886 | </varlistentry> |
| 887 | |
| 888 | <varlistentry> |
| 889 | <term><computeroutput>BZ_RUN_OK, BZ_FLUSH_OK, |
| 890 | BZ_FINISH_OK</computeroutput></term> |
| 891 | <listitem><para>In |
| 892 | <computeroutput>BZ2_bzCompress</computeroutput>, the requested |
| 893 | flush/finish/nothing-special action was completed |
| 894 | successfully.</para></listitem> |
| 895 | </varlistentry> |
| 896 | |
| 897 | <varlistentry> |
| 898 | <term><computeroutput>BZ_STREAM_END</computeroutput></term> |
| 899 | <listitem><para>Compression of data was completed, or the |
| 900 | logical stream end was detected during |
| 901 | decompression.</para></listitem> |
| 902 | </varlistentry> |
| 903 | |
| 904 | </variablelist> |
| 905 | |
| 906 | <para>The following return values indicate an error of some |
| 907 | kind.</para> |
| 908 | |
| 909 | <variablelist> |
| 910 | |
| 911 | <varlistentry> |
| 912 | <term><computeroutput>BZ_CONFIG_ERROR</computeroutput></term> |
| 913 | <listitem><para>Indicates that the library has been improperly |
| 914 | compiled on your platform -- a major configuration error. |
| 915 | Specifically, it means that |
| 916 | <computeroutput>sizeof(char)</computeroutput>, |
| 917 | <computeroutput>sizeof(short)</computeroutput> and |
| 918 | <computeroutput>sizeof(int)</computeroutput> are not 1, 2 and |
| 919 | 4 respectively, as they should be. Note that the library |
| 920 | should still work properly on 64-bit platforms which follow |
| 921 | the LP64 programming model -- that is, where |
| 922 | <computeroutput>sizeof(long)</computeroutput> and |
| 923 | <computeroutput>sizeof(void*)</computeroutput> are 8. Under |
| 924 | LP64, <computeroutput>sizeof(int)</computeroutput> is still 4, |
| 925 | so <computeroutput>libbzip2</computeroutput>, which doesn't |
| 926 | use the <computeroutput>long</computeroutput> type, is |
| 927 | OK.</para></listitem> |
| 928 | </varlistentry> |
| 929 | |
| 930 | <varlistentry> |
| 931 | <term><computeroutput>BZ_SEQUENCE_ERROR</computeroutput></term> |
| 932 | <listitem><para>When using the library, it is important to call |
| 933 | the functions in the correct sequence and with data structures |
| 934 | (buffers etc) in the correct states. |
| 935 | <computeroutput>libbzip2</computeroutput> checks as much as it |
| 936 | can to ensure this is happening, and returns |
| 937 | <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> if not. |
| 938 | Code which complies precisely with the function semantics, as |
| 939 | detailed below, should never receive this value; such an event |
| 940 | denotes buggy code which you should |
| 941 | investigate.</para></listitem> |
| 942 | </varlistentry> |
| 943 | |
| 944 | <varlistentry> |
| 945 | <term><computeroutput>BZ_PARAM_ERROR</computeroutput></term> |
| 946 | <listitem><para>Returned when a parameter to a function call is |
| 947 | out of range or otherwise manifestly incorrect. As with |
| 948 | <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, this |
| 949 | denotes a bug in the client code. The distinction between |
| 950 | <computeroutput>BZ_PARAM_ERROR</computeroutput> and |
| 951 | <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> is a bit |
| 952 | hazy, but still worth making.</para></listitem> |
| 953 | </varlistentry> |
| 954 | |
| 955 | <varlistentry> |
| 956 | <term><computeroutput>BZ_MEM_ERROR</computeroutput></term> |
| 957 | <listitem><para>Returned when a request to allocate memory |
| 958 | failed. Note that the quantity of memory needed to decompress |
| 959 | a stream cannot be determined until the stream's header has |
| 960 | been read. So |
| 961 | <computeroutput>BZ2_bzDecompress</computeroutput> and |
| 962 | <computeroutput>BZ2_bzRead</computeroutput> may return |
| 963 | <computeroutput>BZ_MEM_ERROR</computeroutput> even though some |
| 964 | of the compressed data has been read. The same is not true |
| 965 | for compression; once |
| 966 | <computeroutput>BZ2_bzCompressInit</computeroutput> or |
| 967 | <computeroutput>BZ2_bzWriteOpen</computeroutput> have |
| 968 | successfully completed, |
| 969 | <computeroutput>BZ_MEM_ERROR</computeroutput> cannot |
| 970 | occur.</para></listitem> |
| 971 | </varlistentry> |
| 972 | |
| 973 | <varlistentry> |
| 974 | <term><computeroutput>BZ_DATA_ERROR</computeroutput></term> |
| 975 | <listitem><para>Returned when a data integrity error is |
| 976 | detected during decompression. Most importantly, this means |
| 977 | when stored and computed CRCs for the data do not match. This |
| 978 | value is also returned upon detection of any other anomaly in |
| 979 | the compressed data.</para></listitem> |
| 980 | </varlistentry> |
| 981 | |
| 982 | <varlistentry> |
| 983 | <term><computeroutput>BZ_DATA_ERROR_MAGIC</computeroutput></term> |
| 984 | <listitem><para>As a special case of |
| 985 | <computeroutput>BZ_DATA_ERROR</computeroutput>, it is |
| 986 | sometimes useful to know when the compressed stream does not |
| 987 | start with the correct magic bytes (<computeroutput>'B' 'Z' |
| 988 | 'h'</computeroutput>).</para></listitem> |
| 989 | </varlistentry> |
| 990 | |
| 991 | <varlistentry> |
| 992 | <term><computeroutput>BZ_IO_ERROR</computeroutput></term> |
| 993 | <listitem><para>Returned by |
| 994 | <computeroutput>BZ2_bzRead</computeroutput> and |
| 995 | <computeroutput>BZ2_bzWrite</computeroutput> when there is an |
| 996 | error reading or writing in the compressed file, and by |
| 997 | <computeroutput>BZ2_bzReadOpen</computeroutput> and |
| 998 | <computeroutput>BZ2_bzWriteOpen</computeroutput> for attempts |
| 999 | to use a file for which the error indicator (viz, |
| 1000 | <computeroutput>ferror(f)</computeroutput>) is set. On |
| 1001 | receipt of <computeroutput>BZ_IO_ERROR</computeroutput>, the |
| 1002 | caller should consult <computeroutput>errno</computeroutput> |
| 1003 | and/or <computeroutput>perror</computeroutput> to acquire |
| 1004 | operating-system specific information about the |
| 1005 | problem.</para></listitem> |
| 1006 | </varlistentry> |
| 1007 | |
| 1008 | <varlistentry> |
| 1009 | <term><computeroutput>BZ_UNEXPECTED_EOF</computeroutput></term> |
| 1010 | <listitem><para>Returned by |
| 1011 | <computeroutput>BZ2_bzRead</computeroutput> when the |
| 1012 | compressed file finishes before the logical end of stream is |
| 1013 | detected.</para></listitem> |
| 1014 | </varlistentry> |
| 1015 | |
| 1016 | <varlistentry> |
| 1017 | <term><computeroutput>BZ_OUTBUFF_FULL</computeroutput></term> |
| 1018 | <listitem><para>Returned by |
| 1019 | <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and |
| 1020 | <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> to |
| 1021 | indicate that the output data will not fit into the output |
| 1022 | buffer provided.</para></listitem> |
| 1023 | </varlistentry> |
| 1024 | |
| 1025 | </variablelist> |
| 1026 | |
| 1027 | </sect1> |
| 1028 | |
| 1029 | |
| 1030 | |
| 1031 | <sect1 id="low-level" xreflabel=">Low-level interface"> |
| 1032 | <title>Low-level interface</title> |
| 1033 | |
| 1034 | |
| 1035 | <sect2 id="bzcompress-init" xreflabel="BZ2_bzCompressInit"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1036 | <title>BZ2_bzCompressInit</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1037 | |
| 1038 | <programlisting> |
| 1039 | typedef struct { |
| 1040 | char *next_in; |
| 1041 | unsigned int avail_in; |
| 1042 | unsigned int total_in_lo32; |
| 1043 | unsigned int total_in_hi32; |
| 1044 | |
| 1045 | char *next_out; |
| 1046 | unsigned int avail_out; |
| 1047 | unsigned int total_out_lo32; |
| 1048 | unsigned int total_out_hi32; |
| 1049 | |
| 1050 | void *state; |
| 1051 | |
| 1052 | void *(*bzalloc)(void *,int,int); |
| 1053 | void (*bzfree)(void *,void *); |
| 1054 | void *opaque; |
| 1055 | } bz_stream; |
| 1056 | |
| 1057 | int BZ2_bzCompressInit ( bz_stream *strm, |
| 1058 | int blockSize100k, |
| 1059 | int verbosity, |
| 1060 | int workFactor ); |
| 1061 | </programlisting> |
| 1062 | |
| 1063 | <para>Prepares for compression. The |
| 1064 | <computeroutput>bz_stream</computeroutput> structure holds all |
| 1065 | data pertaining to the compression activity. A |
| 1066 | <computeroutput>bz_stream</computeroutput> structure should be |
| 1067 | allocated and initialised prior to the call. The fields of |
| 1068 | <computeroutput>bz_stream</computeroutput> comprise the entirety |
| 1069 | of the user-visible data. <computeroutput>state</computeroutput> |
| 1070 | is a pointer to the private data structures required for |
| 1071 | compression.</para> |
| 1072 | |
| 1073 | <para>Custom memory allocators are supported, via fields |
| 1074 | <computeroutput>bzalloc</computeroutput>, |
| 1075 | <computeroutput>bzfree</computeroutput>, and |
| 1076 | <computeroutput>opaque</computeroutput>. The value |
| 1077 | <computeroutput>opaque</computeroutput> is passed to as the first |
| 1078 | argument to all calls to <computeroutput>bzalloc</computeroutput> |
| 1079 | and <computeroutput>bzfree</computeroutput>, but is otherwise |
| 1080 | ignored by the library. The call <computeroutput>bzalloc ( |
| 1081 | opaque, n, m )</computeroutput> is expected to return a pointer |
| 1082 | <computeroutput>p</computeroutput> to <computeroutput>n * |
| 1083 | m</computeroutput> bytes of memory, and <computeroutput>bzfree ( |
| 1084 | opaque, p )</computeroutput> should free that memory.</para> |
| 1085 | |
| 1086 | <para>If you don't want to use a custom memory allocator, set |
| 1087 | <computeroutput>bzalloc</computeroutput>, |
| 1088 | <computeroutput>bzfree</computeroutput> and |
| 1089 | <computeroutput>opaque</computeroutput> to |
| 1090 | <computeroutput>NULL</computeroutput>, and the library will then |
| 1091 | use the standard <computeroutput>malloc</computeroutput> / |
| 1092 | <computeroutput>free</computeroutput> routines.</para> |
| 1093 | |
| 1094 | <para>Before calling |
| 1095 | <computeroutput>BZ2_bzCompressInit</computeroutput>, fields |
| 1096 | <computeroutput>bzalloc</computeroutput>, |
| 1097 | <computeroutput>bzfree</computeroutput> and |
| 1098 | <computeroutput>opaque</computeroutput> should be filled |
| 1099 | appropriately, as just described. Upon return, the internal |
| 1100 | state will have been allocated and initialised, and |
| 1101 | <computeroutput>total_in_lo32</computeroutput>, |
| 1102 | <computeroutput>total_in_hi32</computeroutput>, |
| 1103 | <computeroutput>total_out_lo32</computeroutput> and |
| 1104 | <computeroutput>total_out_hi32</computeroutput> will have been |
| 1105 | set to zero. These four fields are used by the library to inform |
| 1106 | the caller of the total amount of data passed into and out of the |
| 1107 | library, respectively. You should not try to change them. As of |
| 1108 | version 1.0, 64-bit counts are maintained, even on 32-bit |
| 1109 | platforms, using the <computeroutput>_hi32</computeroutput> |
| 1110 | fields to store the upper 32 bits of the count. So, for example, |
| 1111 | the total amount of data in is <computeroutput>(total_in_hi32 |
| 1112 | << 32) + total_in_lo32</computeroutput>.</para> |
| 1113 | |
| 1114 | <para>Parameter <computeroutput>blockSize100k</computeroutput> |
| 1115 | specifies the block size to be used for compression. It should |
| 1116 | be a value between 1 and 9 inclusive, and the actual block size |
| 1117 | used is 100000 x this figure. 9 gives the best compression but |
| 1118 | takes most memory.</para> |
| 1119 | |
| 1120 | <para>Parameter <computeroutput>verbosity</computeroutput> should |
| 1121 | be set to a number between 0 and 4 inclusive. 0 is silent, and |
| 1122 | greater numbers give increasingly verbose monitoring/debugging |
| 1123 | output. If the library has been compiled with |
| 1124 | <computeroutput>-DBZ_NO_STDIO</computeroutput>, no such output |
| 1125 | will appear for any verbosity setting.</para> |
| 1126 | |
| 1127 | <para>Parameter <computeroutput>workFactor</computeroutput> |
| 1128 | controls how the compression phase behaves when presented with |
| 1129 | worst case, highly repetitive, input data. If compression runs |
| 1130 | into difficulties caused by repetitive data, the library switches |
| 1131 | from the standard sorting algorithm to a fallback algorithm. The |
| 1132 | fallback is slower than the standard algorithm by perhaps a |
| 1133 | factor of three, but always behaves reasonably, no matter how bad |
| 1134 | the input.</para> |
| 1135 | |
| 1136 | <para>Lower values of <computeroutput>workFactor</computeroutput> |
| 1137 | reduce the amount of effort the standard algorithm will expend |
| 1138 | before resorting to the fallback. You should set this parameter |
| 1139 | carefully; too low, and many inputs will be handled by the |
| 1140 | fallback algorithm and so compress rather slowly, too high, and |
| 1141 | your average-to-worst case compression times can become very |
| 1142 | large. The default value of 30 gives reasonable behaviour over a |
| 1143 | wide range of circumstances.</para> |
| 1144 | |
| 1145 | <para>Allowable values range from 0 to 250 inclusive. 0 is a |
| 1146 | special case, equivalent to using the default value of 30.</para> |
| 1147 | |
| 1148 | <para>Note that the compressed output generated is the same |
| 1149 | regardless of whether or not the fallback algorithm is |
| 1150 | used.</para> |
| 1151 | |
| 1152 | <para>Be aware also that this parameter may disappear entirely in |
| 1153 | future versions of the library. In principle it should be |
| 1154 | possible to devise a good way to automatically choose which |
| 1155 | algorithm to use. Such a mechanism would render the parameter |
| 1156 | obsolete.</para> |
| 1157 | |
| 1158 | <para>Possible return values:</para> |
| 1159 | |
| 1160 | <programlisting> |
| 1161 | BZ_CONFIG_ERROR |
| 1162 | if the library has been mis-compiled |
| 1163 | BZ_PARAM_ERROR |
| 1164 | if strm is NULL |
| 1165 | or blockSize < 1 or blockSize > 9 |
| 1166 | or verbosity < 0 or verbosity > 4 |
| 1167 | or workFactor < 0 or workFactor > 250 |
| 1168 | BZ_MEM_ERROR |
| 1169 | if not enough memory is available |
| 1170 | BZ_OK |
| 1171 | otherwise |
| 1172 | </programlisting> |
| 1173 | |
| 1174 | <para>Allowable next actions:</para> |
| 1175 | |
| 1176 | <programlisting> |
| 1177 | BZ2_bzCompress |
| 1178 | if BZ_OK is returned |
| 1179 | no specific action needed in case of error |
| 1180 | </programlisting> |
| 1181 | |
| 1182 | </sect2> |
| 1183 | |
| 1184 | |
| 1185 | <sect2 id="bzCompress" xreflabel="BZ2_bzCompress"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1186 | <title>BZ2_bzCompress</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1187 | |
| 1188 | <programlisting> |
| 1189 | int BZ2_bzCompress ( bz_stream *strm, int action ); |
| 1190 | </programlisting> |
| 1191 | |
| 1192 | <para>Provides more input and/or output buffer space for the |
| 1193 | library. The caller maintains input and output buffers, and |
| 1194 | calls <computeroutput>BZ2_bzCompress</computeroutput> to transfer |
| 1195 | data between them.</para> |
| 1196 | |
| 1197 | <para>Before each call to |
| 1198 | <computeroutput>BZ2_bzCompress</computeroutput>, |
| 1199 | <computeroutput>next_in</computeroutput> should point at the data |
| 1200 | to be compressed, and <computeroutput>avail_in</computeroutput> |
| 1201 | should indicate how many bytes the library may read. |
| 1202 | <computeroutput>BZ2_bzCompress</computeroutput> updates |
| 1203 | <computeroutput>next_in</computeroutput>, |
| 1204 | <computeroutput>avail_in</computeroutput> and |
| 1205 | <computeroutput>total_in</computeroutput> to reflect the number |
| 1206 | of bytes it has read.</para> |
| 1207 | |
| 1208 | <para>Similarly, <computeroutput>next_out</computeroutput> should |
| 1209 | point to a buffer in which the compressed data is to be placed, |
| 1210 | with <computeroutput>avail_out</computeroutput> indicating how |
| 1211 | much output space is available. |
| 1212 | <computeroutput>BZ2_bzCompress</computeroutput> updates |
| 1213 | <computeroutput>next_out</computeroutput>, |
| 1214 | <computeroutput>avail_out</computeroutput> and |
| 1215 | <computeroutput>total_out</computeroutput> to reflect the number |
| 1216 | of bytes output.</para> |
| 1217 | |
| 1218 | <para>You may provide and remove as little or as much data as you |
| 1219 | like on each call of |
| 1220 | <computeroutput>BZ2_bzCompress</computeroutput>. In the limit, |
| 1221 | it is acceptable to supply and remove data one byte at a time, |
| 1222 | although this would be terribly inefficient. You should always |
| 1223 | ensure that at least one byte of output space is available at |
| 1224 | each call.</para> |
| 1225 | |
| 1226 | <para>A second purpose of |
| 1227 | <computeroutput>BZ2_bzCompress</computeroutput> is to request a |
| 1228 | change of mode of the compressed stream.</para> |
| 1229 | |
| 1230 | <para>Conceptually, a compressed stream can be in one of four |
| 1231 | states: IDLE, RUNNING, FLUSHING and FINISHING. Before |
| 1232 | initialisation |
| 1233 | (<computeroutput>BZ2_bzCompressInit</computeroutput>) and after |
| 1234 | termination (<computeroutput>BZ2_bzCompressEnd</computeroutput>), |
| 1235 | a stream is regarded as IDLE.</para> |
| 1236 | |
| 1237 | <para>Upon initialisation |
| 1238 | (<computeroutput>BZ2_bzCompressInit</computeroutput>), the stream |
| 1239 | is placed in the RUNNING state. Subsequent calls to |
| 1240 | <computeroutput>BZ2_bzCompress</computeroutput> should pass |
| 1241 | <computeroutput>BZ_RUN</computeroutput> as the requested action; |
| 1242 | other actions are illegal and will result in |
| 1243 | <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>.</para> |
| 1244 | |
| 1245 | <para>At some point, the calling program will have provided all |
| 1246 | the input data it wants to. It will then want to finish up -- in |
| 1247 | effect, asking the library to process any data it might have |
| 1248 | buffered internally. In this state, |
| 1249 | <computeroutput>BZ2_bzCompress</computeroutput> will no longer |
| 1250 | attempt to read data from |
| 1251 | <computeroutput>next_in</computeroutput>, but it will want to |
| 1252 | write data to <computeroutput>next_out</computeroutput>. Because |
| 1253 | the output buffer supplied by the user can be arbitrarily small, |
| 1254 | the finishing-up operation cannot necessarily be done with a |
| 1255 | single call of |
| 1256 | <computeroutput>BZ2_bzCompress</computeroutput>.</para> |
| 1257 | |
| 1258 | <para>Instead, the calling program passes |
| 1259 | <computeroutput>BZ_FINISH</computeroutput> as an action to |
| 1260 | <computeroutput>BZ2_bzCompress</computeroutput>. This changes |
| 1261 | the stream's state to FINISHING. Any remaining input (ie, |
| 1262 | <computeroutput>next_in[0 .. avail_in-1]</computeroutput>) is |
| 1263 | compressed and transferred to the output buffer. To do this, |
| 1264 | <computeroutput>BZ2_bzCompress</computeroutput> must be called |
| 1265 | repeatedly until all the output has been consumed. At that |
| 1266 | point, <computeroutput>BZ2_bzCompress</computeroutput> returns |
| 1267 | <computeroutput>BZ_STREAM_END</computeroutput>, and the stream's |
| 1268 | state is set back to IDLE. |
| 1269 | <computeroutput>BZ2_bzCompressEnd</computeroutput> should then be |
| 1270 | called.</para> |
| 1271 | |
| 1272 | <para>Just to make sure the calling program does not cheat, the |
| 1273 | library makes a note of <computeroutput>avail_in</computeroutput> |
| 1274 | at the time of the first call to |
| 1275 | <computeroutput>BZ2_bzCompress</computeroutput> which has |
| 1276 | <computeroutput>BZ_FINISH</computeroutput> as an action (ie, at |
| 1277 | the time the program has announced its intention to not supply |
| 1278 | any more input). By comparing this value with that of |
| 1279 | <computeroutput>avail_in</computeroutput> over subsequent calls |
| 1280 | to <computeroutput>BZ2_bzCompress</computeroutput>, the library |
| 1281 | can detect any attempts to slip in more data to compress. Any |
| 1282 | calls for which this is detected will return |
| 1283 | <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>. This |
| 1284 | indicates a programming mistake which should be corrected.</para> |
| 1285 | |
| 1286 | <para>Instead of asking to finish, the calling program may ask |
| 1287 | <computeroutput>BZ2_bzCompress</computeroutput> to take all the |
| 1288 | remaining input, compress it and terminate the current |
| 1289 | (Burrows-Wheeler) compression block. This could be useful for |
| 1290 | error control purposes. The mechanism is analogous to that for |
| 1291 | finishing: call <computeroutput>BZ2_bzCompress</computeroutput> |
| 1292 | with an action of <computeroutput>BZ_FLUSH</computeroutput>, |
| 1293 | remove output data, and persist with the |
| 1294 | <computeroutput>BZ_FLUSH</computeroutput> action until the value |
| 1295 | <computeroutput>BZ_RUN</computeroutput> is returned. As with |
| 1296 | finishing, <computeroutput>BZ2_bzCompress</computeroutput> |
| 1297 | detects any attempt to provide more input data once the flush has |
| 1298 | begun.</para> |
| 1299 | |
| 1300 | <para>Once the flush is complete, the stream returns to the |
| 1301 | normal RUNNING state.</para> |
| 1302 | |
| 1303 | <para>This all sounds pretty complex, but isn't really. Here's a |
| 1304 | table which shows which actions are allowable in each state, what |
| 1305 | action will be taken, what the next state is, and what the |
| 1306 | non-error return values are. Note that you can't explicitly ask |
| 1307 | what state the stream is in, but nor do you need to -- it can be |
| 1308 | inferred from the values returned by |
| 1309 | <computeroutput>BZ2_bzCompress</computeroutput>.</para> |
| 1310 | |
| 1311 | <programlisting> |
| 1312 | IDLE/any |
| 1313 | Illegal. IDLE state only exists after BZ2_bzCompressEnd or |
| 1314 | before BZ2_bzCompressInit. |
| 1315 | Return value = BZ_SEQUENCE_ERROR |
| 1316 | |
| 1317 | RUNNING/BZ_RUN |
| 1318 | Compress from next_in to next_out as much as possible. |
| 1319 | Next state = RUNNING |
| 1320 | Return value = BZ_RUN_OK |
| 1321 | |
| 1322 | RUNNING/BZ_FLUSH |
| 1323 | Remember current value of next_in. Compress from next_in |
| 1324 | to next_out as much as possible, but do not accept any more input. |
| 1325 | Next state = FLUSHING |
| 1326 | Return value = BZ_FLUSH_OK |
| 1327 | |
| 1328 | RUNNING/BZ_FINISH |
| 1329 | Remember current value of next_in. Compress from next_in |
| 1330 | to next_out as much as possible, but do not accept any more input. |
| 1331 | Next state = FINISHING |
| 1332 | Return value = BZ_FINISH_OK |
| 1333 | |
| 1334 | FLUSHING/BZ_FLUSH |
| 1335 | Compress from next_in to next_out as much as possible, |
| 1336 | but do not accept any more input. |
| 1337 | If all the existing input has been used up and all compressed |
| 1338 | output has been removed |
| 1339 | Next state = RUNNING; Return value = BZ_RUN_OK |
| 1340 | else |
| 1341 | Next state = FLUSHING; Return value = BZ_FLUSH_OK |
| 1342 | |
| 1343 | FLUSHING/other |
| 1344 | Illegal. |
| 1345 | Return value = BZ_SEQUENCE_ERROR |
| 1346 | |
| 1347 | FINISHING/BZ_FINISH |
| 1348 | Compress from next_in to next_out as much as possible, |
| 1349 | but to not accept any more input. |
| 1350 | If all the existing input has been used up and all compressed |
| 1351 | output has been removed |
| 1352 | Next state = IDLE; Return value = BZ_STREAM_END |
| 1353 | else |
| 1354 | Next state = FINISHING; Return value = BZ_FINISH_OK |
| 1355 | |
| 1356 | FINISHING/other |
| 1357 | Illegal. |
| 1358 | Return value = BZ_SEQUENCE_ERROR |
| 1359 | </programlisting> |
| 1360 | |
| 1361 | |
| 1362 | <para>That still looks complicated? Well, fair enough. The |
| 1363 | usual sequence of calls for compressing a load of data is:</para> |
| 1364 | |
| 1365 | <orderedlist> |
| 1366 | |
| 1367 | <listitem><para>Get started with |
| 1368 | <computeroutput>BZ2_bzCompressInit</computeroutput>.</para></listitem> |
| 1369 | |
| 1370 | <listitem><para>Shovel data in and shlurp out its compressed form |
| 1371 | using zero or more calls of |
| 1372 | <computeroutput>BZ2_bzCompress</computeroutput> with action = |
| 1373 | <computeroutput>BZ_RUN</computeroutput>.</para></listitem> |
| 1374 | |
| 1375 | <listitem><para>Finish up. Repeatedly call |
| 1376 | <computeroutput>BZ2_bzCompress</computeroutput> with action = |
| 1377 | <computeroutput>BZ_FINISH</computeroutput>, copying out the |
| 1378 | compressed output, until |
| 1379 | <computeroutput>BZ_STREAM_END</computeroutput> is |
| 1380 | returned.</para></listitem> <listitem><para>Close up and go home. Call |
| 1381 | <computeroutput>BZ2_bzCompressEnd</computeroutput>.</para></listitem> |
| 1382 | |
| 1383 | </orderedlist> |
| 1384 | |
| 1385 | <para>If the data you want to compress fits into your input |
| 1386 | buffer all at once, you can skip the calls of |
| 1387 | <computeroutput>BZ2_bzCompress ( ..., BZ_RUN )</computeroutput> |
| 1388 | and just do the <computeroutput>BZ2_bzCompress ( ..., BZ_FINISH |
| 1389 | )</computeroutput> calls.</para> |
| 1390 | |
| 1391 | <para>All required memory is allocated by |
| 1392 | <computeroutput>BZ2_bzCompressInit</computeroutput>. The |
| 1393 | compression library can accept any data at all (obviously). So |
| 1394 | you shouldn't get any error return values from the |
| 1395 | <computeroutput>BZ2_bzCompress</computeroutput> calls. If you |
| 1396 | do, they will be |
| 1397 | <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, and indicate |
| 1398 | a bug in your programming.</para> |
| 1399 | |
| 1400 | <para>Trivial other possible return values:</para> |
| 1401 | |
| 1402 | <programlisting> |
| 1403 | BZ_PARAM_ERROR |
| 1404 | if strm is NULL, or strm->s is NULL |
| 1405 | </programlisting> |
| 1406 | |
| 1407 | </sect2> |
| 1408 | |
| 1409 | |
| 1410 | <sect2 id="bzCompress-end" xreflabel="BZ2_bzCompressEnd"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1411 | <title>BZ2_bzCompressEnd</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1412 | |
| 1413 | <programlisting> |
| 1414 | int BZ2_bzCompressEnd ( bz_stream *strm ); |
| 1415 | </programlisting> |
| 1416 | |
| 1417 | <para>Releases all memory associated with a compression |
| 1418 | stream.</para> |
| 1419 | |
| 1420 | <para>Possible return values:</para> |
| 1421 | |
| 1422 | <programlisting> |
| 1423 | BZ_PARAM_ERROR if strm is NULL or strm->s is NULL |
| 1424 | BZ_OK otherwise |
| 1425 | </programlisting> |
| 1426 | |
| 1427 | </sect2> |
| 1428 | |
| 1429 | |
| 1430 | <sect2 id="bzDecompress-init" xreflabel="BZ2_bzDecompressInit"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1431 | <title>BZ2_bzDecompressInit</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1432 | |
| 1433 | <programlisting> |
| 1434 | int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small ); |
| 1435 | </programlisting> |
| 1436 | |
| 1437 | <para>Prepares for decompression. As with |
| 1438 | <computeroutput>BZ2_bzCompressInit</computeroutput>, a |
| 1439 | <computeroutput>bz_stream</computeroutput> record should be |
| 1440 | allocated and initialised before the call. Fields |
| 1441 | <computeroutput>bzalloc</computeroutput>, |
| 1442 | <computeroutput>bzfree</computeroutput> and |
| 1443 | <computeroutput>opaque</computeroutput> should be set if a custom |
| 1444 | memory allocator is required, or made |
| 1445 | <computeroutput>NULL</computeroutput> for the normal |
| 1446 | <computeroutput>malloc</computeroutput> / |
| 1447 | <computeroutput>free</computeroutput> routines. Upon return, the |
| 1448 | internal state will have been initialised, and |
| 1449 | <computeroutput>total_in</computeroutput> and |
| 1450 | <computeroutput>total_out</computeroutput> will be zero.</para> |
| 1451 | |
| 1452 | <para>For the meaning of parameter |
| 1453 | <computeroutput>verbosity</computeroutput>, see |
| 1454 | <computeroutput>BZ2_bzCompressInit</computeroutput>.</para> |
| 1455 | |
| 1456 | <para>If <computeroutput>small</computeroutput> is nonzero, the |
| 1457 | library will use an alternative decompression algorithm which |
| 1458 | uses less memory but at the cost of decompressing more slowly |
| 1459 | (roughly speaking, half the speed, but the maximum memory |
| 1460 | requirement drops to around 2300k). See <xref linkend="using"/> |
| 1461 | for more information on memory management.</para> |
| 1462 | |
| 1463 | <para>Note that the amount of memory needed to decompress a |
| 1464 | stream cannot be determined until the stream's header has been |
| 1465 | read, so even if |
| 1466 | <computeroutput>BZ2_bzDecompressInit</computeroutput> succeeds, a |
| 1467 | subsequent <computeroutput>BZ2_bzDecompress</computeroutput> |
| 1468 | could fail with |
| 1469 | <computeroutput>BZ_MEM_ERROR</computeroutput>.</para> |
| 1470 | |
| 1471 | <para>Possible return values:</para> |
| 1472 | |
| 1473 | <programlisting> |
| 1474 | BZ_CONFIG_ERROR |
| 1475 | if the library has been mis-compiled |
| 1476 | BZ_PARAM_ERROR |
| 1477 | if ( small != 0 && small != 1 ) |
| 1478 | or (verbosity <; 0 || verbosity > 4) |
| 1479 | BZ_MEM_ERROR |
| 1480 | if insufficient memory is available |
| 1481 | </programlisting> |
| 1482 | |
| 1483 | <para>Allowable next actions:</para> |
| 1484 | |
| 1485 | <programlisting> |
| 1486 | BZ2_bzDecompress |
| 1487 | if BZ_OK was returned |
| 1488 | no specific action required in case of error |
| 1489 | </programlisting> |
| 1490 | |
| 1491 | </sect2> |
| 1492 | |
| 1493 | |
| 1494 | <sect2 id="bzDecompress" xreflabel="BZ2_bzDecompress"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1495 | <title>BZ2_bzDecompress</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1496 | |
| 1497 | <programlisting> |
| 1498 | int BZ2_bzDecompress ( bz_stream *strm ); |
| 1499 | </programlisting> |
| 1500 | |
| 1501 | <para>Provides more input and/out output buffer space for the |
| 1502 | library. The caller maintains input and output buffers, and uses |
| 1503 | <computeroutput>BZ2_bzDecompress</computeroutput> to transfer |
| 1504 | data between them.</para> |
| 1505 | |
| 1506 | <para>Before each call to |
| 1507 | <computeroutput>BZ2_bzDecompress</computeroutput>, |
| 1508 | <computeroutput>next_in</computeroutput> should point at the |
| 1509 | compressed data, and <computeroutput>avail_in</computeroutput> |
| 1510 | should indicate how many bytes the library may read. |
| 1511 | <computeroutput>BZ2_bzDecompress</computeroutput> updates |
| 1512 | <computeroutput>next_in</computeroutput>, |
| 1513 | <computeroutput>avail_in</computeroutput> and |
| 1514 | <computeroutput>total_in</computeroutput> to reflect the number |
| 1515 | of bytes it has read.</para> |
| 1516 | |
| 1517 | <para>Similarly, <computeroutput>next_out</computeroutput> should |
| 1518 | point to a buffer in which the uncompressed output is to be |
| 1519 | placed, with <computeroutput>avail_out</computeroutput> |
| 1520 | indicating how much output space is available. |
| 1521 | <computeroutput>BZ2_bzCompress</computeroutput> updates |
| 1522 | <computeroutput>next_out</computeroutput>, |
| 1523 | <computeroutput>avail_out</computeroutput> and |
| 1524 | <computeroutput>total_out</computeroutput> to reflect the number |
| 1525 | of bytes output.</para> |
| 1526 | |
| 1527 | <para>You may provide and remove as little or as much data as you |
| 1528 | like on each call of |
| 1529 | <computeroutput>BZ2_bzDecompress</computeroutput>. In the limit, |
| 1530 | it is acceptable to supply and remove data one byte at a time, |
| 1531 | although this would be terribly inefficient. You should always |
| 1532 | ensure that at least one byte of output space is available at |
| 1533 | each call.</para> |
| 1534 | |
| 1535 | <para>Use of <computeroutput>BZ2_bzDecompress</computeroutput> is |
| 1536 | simpler than |
| 1537 | <computeroutput>BZ2_bzCompress</computeroutput>.</para> |
| 1538 | |
| 1539 | <para>You should provide input and remove output as described |
| 1540 | above, and repeatedly call |
| 1541 | <computeroutput>BZ2_bzDecompress</computeroutput> until |
| 1542 | <computeroutput>BZ_STREAM_END</computeroutput> is returned. |
| 1543 | Appearance of <computeroutput>BZ_STREAM_END</computeroutput> |
| 1544 | denotes that <computeroutput>BZ2_bzDecompress</computeroutput> |
| 1545 | has detected the logical end of the compressed stream. |
| 1546 | <computeroutput>BZ2_bzDecompress</computeroutput> will not |
| 1547 | produce <computeroutput>BZ_STREAM_END</computeroutput> until all |
| 1548 | output data has been placed into the output buffer, so once |
| 1549 | <computeroutput>BZ_STREAM_END</computeroutput> appears, you are |
| 1550 | guaranteed to have available all the decompressed output, and |
| 1551 | <computeroutput>BZ2_bzDecompressEnd</computeroutput> can safely |
| 1552 | be called.</para> |
| 1553 | |
| 1554 | <para>If case of an error return value, you should call |
| 1555 | <computeroutput>BZ2_bzDecompressEnd</computeroutput> to clean up |
| 1556 | and release memory.</para> |
| 1557 | |
| 1558 | <para>Possible return values:</para> |
| 1559 | |
| 1560 | <programlisting> |
| 1561 | BZ_PARAM_ERROR |
| 1562 | if strm is NULL or strm->s is NULL |
| 1563 | or strm->avail_out < 1 |
| 1564 | BZ_DATA_ERROR |
| 1565 | if a data integrity error is detected in the compressed stream |
| 1566 | BZ_DATA_ERROR_MAGIC |
| 1567 | if the compressed stream doesn't begin with the right magic bytes |
| 1568 | BZ_MEM_ERROR |
| 1569 | if there wasn't enough memory available |
| 1570 | BZ_STREAM_END |
| 1571 | if the logical end of the data stream was detected and all |
| 1572 | output in has been consumed, eg s-->avail_out > 0 |
| 1573 | BZ_OK |
| 1574 | otherwise |
| 1575 | </programlisting> |
| 1576 | |
| 1577 | <para>Allowable next actions:</para> |
| 1578 | |
| 1579 | <programlisting> |
| 1580 | BZ2_bzDecompress |
| 1581 | if BZ_OK was returned |
| 1582 | BZ2_bzDecompressEnd |
| 1583 | otherwise |
| 1584 | </programlisting> |
| 1585 | |
| 1586 | </sect2> |
| 1587 | |
| 1588 | |
| 1589 | <sect2 id="bzDecompress-end" xreflabel="BZ2_bzDecompressEnd"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1590 | <title>BZ2_bzDecompressEnd</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1591 | |
| 1592 | <programlisting> |
| 1593 | int BZ2_bzDecompressEnd ( bz_stream *strm ); |
| 1594 | </programlisting> |
| 1595 | |
| 1596 | <para>Releases all memory associated with a decompression |
| 1597 | stream.</para> |
| 1598 | |
| 1599 | <para>Possible return values:</para> |
| 1600 | |
| 1601 | <programlisting> |
| 1602 | BZ_PARAM_ERROR |
| 1603 | if strm is NULL or strm->s is NULL |
| 1604 | BZ_OK |
| 1605 | otherwise |
| 1606 | </programlisting> |
| 1607 | |
| 1608 | <para>Allowable next actions:</para> |
| 1609 | |
| 1610 | <programlisting> |
| 1611 | None. |
| 1612 | </programlisting> |
| 1613 | |
| 1614 | </sect2> |
| 1615 | |
| 1616 | </sect1> |
| 1617 | |
| 1618 | |
| 1619 | <sect1 id="hl-interface" xreflabel="High-level interface"> |
| 1620 | <title>High-level interface</title> |
| 1621 | |
| 1622 | <para>This interface provides functions for reading and writing |
| 1623 | <computeroutput>bzip2</computeroutput> format files. First, some |
| 1624 | general points.</para> |
| 1625 | |
| 1626 | <itemizedlist mark='bullet'> |
| 1627 | |
| 1628 | <listitem><para>All of the functions take an |
| 1629 | <computeroutput>int*</computeroutput> first argument, |
| 1630 | <computeroutput>bzerror</computeroutput>. After each call, |
| 1631 | <computeroutput>bzerror</computeroutput> should be consulted |
| 1632 | first to determine the outcome of the call. If |
| 1633 | <computeroutput>bzerror</computeroutput> is |
| 1634 | <computeroutput>BZ_OK</computeroutput>, the call completed |
| 1635 | successfully, and only then should the return value of the |
| 1636 | function (if any) be consulted. If |
| 1637 | <computeroutput>bzerror</computeroutput> is |
| 1638 | <computeroutput>BZ_IO_ERROR</computeroutput>, there was an |
| 1639 | error reading/writing the underlying compressed file, and you |
| 1640 | should then consult <computeroutput>errno</computeroutput> / |
| 1641 | <computeroutput>perror</computeroutput> to determine the cause |
| 1642 | of the difficulty. <computeroutput>bzerror</computeroutput> |
| 1643 | may also be set to various other values; precise details are |
| 1644 | given on a per-function basis below.</para></listitem> |
| 1645 | |
| 1646 | <listitem><para>If <computeroutput>bzerror</computeroutput> indicates |
| 1647 | an error (ie, anything except |
| 1648 | <computeroutput>BZ_OK</computeroutput> and |
| 1649 | <computeroutput>BZ_STREAM_END</computeroutput>), you should |
| 1650 | immediately call |
| 1651 | <computeroutput>BZ2_bzReadClose</computeroutput> (or |
| 1652 | <computeroutput>BZ2_bzWriteClose</computeroutput>, depending on |
| 1653 | whether you are attempting to read or to write) to free up all |
| 1654 | resources associated with the stream. Once an error has been |
| 1655 | indicated, behaviour of all calls except |
| 1656 | <computeroutput>BZ2_bzReadClose</computeroutput> |
| 1657 | (<computeroutput>BZ2_bzWriteClose</computeroutput>) is |
| 1658 | undefined. The implication is that (1) |
| 1659 | <computeroutput>bzerror</computeroutput> should be checked |
| 1660 | after each call, and (2) if |
| 1661 | <computeroutput>bzerror</computeroutput> indicates an error, |
| 1662 | <computeroutput>BZ2_bzReadClose</computeroutput> |
| 1663 | (<computeroutput>BZ2_bzWriteClose</computeroutput>) should then |
| 1664 | be called to clean up.</para></listitem> |
| 1665 | |
| 1666 | <listitem><para>The <computeroutput>FILE*</computeroutput> arguments |
| 1667 | passed to <computeroutput>BZ2_bzReadOpen</computeroutput> / |
| 1668 | <computeroutput>BZ2_bzWriteOpen</computeroutput> should be set |
| 1669 | to binary mode. Most Unix systems will do this by default, but |
| 1670 | other platforms, including Windows and Mac, will not. If you |
| 1671 | omit this, you may encounter problems when moving code to new |
| 1672 | platforms.</para></listitem> |
| 1673 | |
| 1674 | <listitem><para>Memory allocation requests are handled by |
| 1675 | <computeroutput>malloc</computeroutput> / |
| 1676 | <computeroutput>free</computeroutput>. At present there is no |
| 1677 | facility for user-defined memory allocators in the file I/O |
| 1678 | functions (could easily be added, though).</para></listitem> |
| 1679 | |
| 1680 | </itemizedlist> |
| 1681 | |
| 1682 | |
| 1683 | |
| 1684 | <sect2 id="bzreadopen" xreflabel="BZ2_bzReadOpen"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1685 | <title>BZ2_bzReadOpen</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1686 | |
| 1687 | <programlisting> |
| 1688 | typedef void BZFILE; |
| 1689 | |
| 1690 | BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f, |
| 1691 | int verbosity, int small, |
| 1692 | void *unused, int nUnused ); |
| 1693 | </programlisting> |
| 1694 | |
| 1695 | <para>Prepare to read compressed data from file handle |
| 1696 | <computeroutput>f</computeroutput>. |
| 1697 | <computeroutput>f</computeroutput> should refer to a file which |
| 1698 | has been opened for reading, and for which the error indicator |
| 1699 | (<computeroutput>ferror(f)</computeroutput>)is not set. If |
| 1700 | <computeroutput>small</computeroutput> is 1, the library will try |
| 1701 | to decompress using less memory, at the expense of speed.</para> |
| 1702 | |
| 1703 | <para>For reasons explained below, |
| 1704 | <computeroutput>BZ2_bzRead</computeroutput> will decompress the |
| 1705 | <computeroutput>nUnused</computeroutput> bytes starting at |
| 1706 | <computeroutput>unused</computeroutput>, before starting to read |
| 1707 | from the file <computeroutput>f</computeroutput>. At most |
| 1708 | <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes may be |
| 1709 | supplied like this. If this facility is not required, you should |
| 1710 | pass <computeroutput>NULL</computeroutput> and |
| 1711 | <computeroutput>0</computeroutput> for |
| 1712 | <computeroutput>unused</computeroutput> and |
| 1713 | n<computeroutput>Unused</computeroutput> respectively.</para> |
| 1714 | |
| 1715 | <para>For the meaning of parameters |
| 1716 | <computeroutput>small</computeroutput> and |
| 1717 | <computeroutput>verbosity</computeroutput>, see |
| 1718 | <computeroutput>BZ2_bzDecompressInit</computeroutput>.</para> |
| 1719 | |
| 1720 | <para>The amount of memory needed to decompress a file cannot be |
| 1721 | determined until the file's header has been read. So it is |
| 1722 | possible that <computeroutput>BZ2_bzReadOpen</computeroutput> |
| 1723 | returns <computeroutput>BZ_OK</computeroutput> but a subsequent |
| 1724 | call of <computeroutput>BZ2_bzRead</computeroutput> will return |
| 1725 | <computeroutput>BZ_MEM_ERROR</computeroutput>.</para> |
| 1726 | |
| 1727 | <para>Possible assignments to |
| 1728 | <computeroutput>bzerror</computeroutput>:</para> |
| 1729 | |
| 1730 | <programlisting> |
| 1731 | BZ_CONFIG_ERROR |
| 1732 | if the library has been mis-compiled |
| 1733 | BZ_PARAM_ERROR |
| 1734 | if f is NULL |
| 1735 | or small is neither 0 nor 1 |
| 1736 | or ( unused == NULL && nUnused != 0 ) |
| 1737 | or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) ) |
| 1738 | BZ_IO_ERROR |
| 1739 | if ferror(f) is nonzero |
| 1740 | BZ_MEM_ERROR |
| 1741 | if insufficient memory is available |
| 1742 | BZ_OK |
| 1743 | otherwise. |
| 1744 | </programlisting> |
| 1745 | |
| 1746 | <para>Possible return values:</para> |
| 1747 | |
| 1748 | <programlisting> |
| 1749 | Pointer to an abstract BZFILE |
| 1750 | if bzerror is BZ_OK |
| 1751 | NULL |
| 1752 | otherwise |
| 1753 | </programlisting> |
| 1754 | |
| 1755 | <para>Allowable next actions:</para> |
| 1756 | |
| 1757 | <programlisting> |
| 1758 | BZ2_bzRead |
| 1759 | if bzerror is BZ_OK |
| 1760 | BZ2_bzClose |
| 1761 | otherwise |
| 1762 | </programlisting> |
| 1763 | |
| 1764 | </sect2> |
| 1765 | |
| 1766 | |
| 1767 | <sect2 id="bzread" xreflabel="BZ2_bzRead"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1768 | <title>BZ2_bzRead</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1769 | |
| 1770 | <programlisting> |
| 1771 | int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len ); |
| 1772 | </programlisting> |
| 1773 | |
| 1774 | <para>Reads up to <computeroutput>len</computeroutput> |
| 1775 | (uncompressed) bytes from the compressed file |
| 1776 | <computeroutput>b</computeroutput> into the buffer |
| 1777 | <computeroutput>buf</computeroutput>. If the read was |
| 1778 | successful, <computeroutput>bzerror</computeroutput> is set to |
| 1779 | <computeroutput>BZ_OK</computeroutput> and the number of bytes |
| 1780 | read is returned. If the logical end-of-stream was detected, |
| 1781 | <computeroutput>bzerror</computeroutput> will be set to |
| 1782 | <computeroutput>BZ_STREAM_END</computeroutput>, and the number of |
| 1783 | bytes read is returned. All other |
| 1784 | <computeroutput>bzerror</computeroutput> values denote an |
| 1785 | error.</para> |
| 1786 | |
| 1787 | <para><computeroutput>BZ2_bzRead</computeroutput> will supply |
| 1788 | <computeroutput>len</computeroutput> bytes, unless the logical |
| 1789 | stream end is detected or an error occurs. Because of this, it |
| 1790 | is possible to detect the stream end by observing when the number |
| 1791 | of bytes returned is less than the number requested. |
| 1792 | Nevertheless, this is regarded as inadvisable; you should instead |
| 1793 | check <computeroutput>bzerror</computeroutput> after every call |
| 1794 | and watch out for |
| 1795 | <computeroutput>BZ_STREAM_END</computeroutput>.</para> |
| 1796 | |
| 1797 | <para>Internally, <computeroutput>BZ2_bzRead</computeroutput> |
| 1798 | copies data from the compressed file in chunks of size |
| 1799 | <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes before |
| 1800 | decompressing it. If the file contains more bytes than strictly |
| 1801 | needed to reach the logical end-of-stream, |
| 1802 | <computeroutput>BZ2_bzRead</computeroutput> will almost certainly |
| 1803 | read some of the trailing data before signalling |
| 1804 | <computeroutput>BZ_SEQUENCE_END</computeroutput>. To collect the |
| 1805 | read but unused data once |
| 1806 | <computeroutput>BZ_SEQUENCE_END</computeroutput> has appeared, |
| 1807 | call <computeroutput>BZ2_bzReadGetUnused</computeroutput> |
| 1808 | immediately before |
| 1809 | <computeroutput>BZ2_bzReadClose</computeroutput>.</para> |
| 1810 | |
| 1811 | <para>Possible assignments to |
| 1812 | <computeroutput>bzerror</computeroutput>:</para> |
| 1813 | |
| 1814 | <programlisting> |
| 1815 | BZ_PARAM_ERROR |
| 1816 | if b is NULL or buf is NULL or len < 0 |
| 1817 | BZ_SEQUENCE_ERROR |
| 1818 | if b was opened with BZ2_bzWriteOpen |
| 1819 | BZ_IO_ERROR |
| 1820 | if there is an error reading from the compressed file |
| 1821 | BZ_UNEXPECTED_EOF |
| 1822 | if the compressed file ended before |
| 1823 | the logical end-of-stream was detected |
| 1824 | BZ_DATA_ERROR |
| 1825 | if a data integrity error was detected in the compressed stream |
| 1826 | BZ_DATA_ERROR_MAGIC |
| 1827 | if the stream does not begin with the requisite header bytes |
| 1828 | (ie, is not a bzip2 data file). This is really |
| 1829 | a special case of BZ_DATA_ERROR. |
| 1830 | BZ_MEM_ERROR |
| 1831 | if insufficient memory was available |
| 1832 | BZ_STREAM_END |
| 1833 | if the logical end of stream was detected. |
| 1834 | BZ_OK |
| 1835 | otherwise. |
| 1836 | </programlisting> |
| 1837 | |
| 1838 | <para>Possible return values:</para> |
| 1839 | |
| 1840 | <programlisting> |
| 1841 | number of bytes read |
| 1842 | if bzerror is BZ_OK or BZ_STREAM_END |
| 1843 | undefined |
| 1844 | otherwise |
| 1845 | </programlisting> |
| 1846 | |
| 1847 | <para>Allowable next actions:</para> |
| 1848 | |
| 1849 | <programlisting> |
| 1850 | collect data from buf, then BZ2_bzRead or BZ2_bzReadClose |
| 1851 | if bzerror is BZ_OK |
| 1852 | collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused |
| 1853 | if bzerror is BZ_SEQUENCE_END |
| 1854 | BZ2_bzReadClose |
| 1855 | otherwise |
| 1856 | </programlisting> |
| 1857 | |
| 1858 | </sect2> |
| 1859 | |
| 1860 | |
| 1861 | <sect2 id="bzreadgetunused" xreflabel="BZ2_bzReadGetUnused"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1862 | <title>BZ2_bzReadGetUnused</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1863 | |
| 1864 | <programlisting> |
| 1865 | void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b, |
| 1866 | void** unused, int* nUnused ); |
| 1867 | </programlisting> |
| 1868 | |
| 1869 | <para>Returns data which was read from the compressed file but |
| 1870 | was not needed to get to the logical end-of-stream. |
| 1871 | <computeroutput>*unused</computeroutput> is set to the address of |
| 1872 | the data, and <computeroutput>*nUnused</computeroutput> to the |
| 1873 | number of bytes. <computeroutput>*nUnused</computeroutput> will |
| 1874 | be set to a value between <computeroutput>0</computeroutput> and |
| 1875 | <computeroutput>BZ_MAX_UNUSED</computeroutput> inclusive.</para> |
| 1876 | |
| 1877 | <para>This function may only be called once |
| 1878 | <computeroutput>BZ2_bzRead</computeroutput> has signalled |
| 1879 | <computeroutput>BZ_STREAM_END</computeroutput> but before |
| 1880 | <computeroutput>BZ2_bzReadClose</computeroutput>.</para> |
| 1881 | |
| 1882 | <para>Possible assignments to |
| 1883 | <computeroutput>bzerror</computeroutput>:</para> |
| 1884 | |
| 1885 | <programlisting> |
| 1886 | BZ_PARAM_ERROR |
| 1887 | if b is NULL |
| 1888 | or unused is NULL or nUnused is NULL |
| 1889 | BZ_SEQUENCE_ERROR |
| 1890 | if BZ_STREAM_END has not been signalled |
| 1891 | or if b was opened with BZ2_bzWriteOpen |
| 1892 | BZ_OK |
| 1893 | otherwise |
| 1894 | </programlisting> |
| 1895 | |
| 1896 | <para>Allowable next actions:</para> |
| 1897 | |
| 1898 | <programlisting> |
| 1899 | BZ2_bzReadClose |
| 1900 | </programlisting> |
| 1901 | |
| 1902 | </sect2> |
| 1903 | |
| 1904 | |
| 1905 | <sect2 id="bzreadclose" xreflabel="BZ2_bzReadClose"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1906 | <title>BZ2_bzReadClose</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1907 | |
| 1908 | <programlisting> |
| 1909 | void BZ2_bzReadClose ( int *bzerror, BZFILE *b ); |
| 1910 | </programlisting> |
| 1911 | |
| 1912 | <para>Releases all memory pertaining to the compressed file |
| 1913 | <computeroutput>b</computeroutput>. |
| 1914 | <computeroutput>BZ2_bzReadClose</computeroutput> does not call |
| 1915 | <computeroutput>fclose</computeroutput> on the underlying file |
| 1916 | handle, so you should do that yourself if appropriate. |
| 1917 | <computeroutput>BZ2_bzReadClose</computeroutput> should be called |
| 1918 | to clean up after all error situations.</para> |
| 1919 | |
| 1920 | <para>Possible assignments to |
| 1921 | <computeroutput>bzerror</computeroutput>:</para> |
| 1922 | |
| 1923 | <programlisting> |
| 1924 | BZ_SEQUENCE_ERROR |
| 1925 | if b was opened with BZ2_bzOpenWrite |
| 1926 | BZ_OK |
| 1927 | otherwise |
| 1928 | </programlisting> |
| 1929 | |
| 1930 | <para>Allowable next actions:</para> |
| 1931 | |
| 1932 | <programlisting> |
| 1933 | none |
| 1934 | </programlisting> |
| 1935 | |
| 1936 | </sect2> |
| 1937 | |
| 1938 | |
| 1939 | <sect2 id="bzwriteopen" xreflabel="BZ2_bzWriteOpen"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 1940 | <title>BZ2_bzWriteOpen</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 1941 | |
| 1942 | <programlisting> |
| 1943 | BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f, |
| 1944 | int blockSize100k, int verbosity, |
| 1945 | int workFactor ); |
| 1946 | </programlisting> |
| 1947 | |
| 1948 | <para>Prepare to write compressed data to file handle |
| 1949 | <computeroutput>f</computeroutput>. |
| 1950 | <computeroutput>f</computeroutput> should refer to a file which |
| 1951 | has been opened for writing, and for which the error indicator |
| 1952 | (<computeroutput>ferror(f)</computeroutput>)is not set.</para> |
| 1953 | |
| 1954 | <para>For the meaning of parameters |
| 1955 | <computeroutput>blockSize100k</computeroutput>, |
| 1956 | <computeroutput>verbosity</computeroutput> and |
| 1957 | <computeroutput>workFactor</computeroutput>, see |
| 1958 | <computeroutput>BZ2_bzCompressInit</computeroutput>.</para> |
| 1959 | |
| 1960 | <para>All required memory is allocated at this stage, so if the |
| 1961 | call completes successfully, |
| 1962 | <computeroutput>BZ_MEM_ERROR</computeroutput> cannot be signalled |
| 1963 | by a subsequent call to |
| 1964 | <computeroutput>BZ2_bzWrite</computeroutput>.</para> |
| 1965 | |
| 1966 | <para>Possible assignments to |
| 1967 | <computeroutput>bzerror</computeroutput>:</para> |
| 1968 | |
| 1969 | <programlisting> |
| 1970 | BZ_CONFIG_ERROR |
| 1971 | if the library has been mis-compiled |
| 1972 | BZ_PARAM_ERROR |
| 1973 | if f is NULL |
| 1974 | or blockSize100k < 1 or blockSize100k > 9 |
| 1975 | BZ_IO_ERROR |
| 1976 | if ferror(f) is nonzero |
| 1977 | BZ_MEM_ERROR |
| 1978 | if insufficient memory is available |
| 1979 | BZ_OK |
| 1980 | otherwise |
| 1981 | </programlisting> |
| 1982 | |
| 1983 | <para>Possible return values:</para> |
| 1984 | |
| 1985 | <programlisting> |
| 1986 | Pointer to an abstract BZFILE |
| 1987 | if bzerror is BZ_OK |
| 1988 | NULL |
| 1989 | otherwise |
| 1990 | </programlisting> |
| 1991 | |
| 1992 | <para>Allowable next actions:</para> |
| 1993 | |
| 1994 | <programlisting> |
| 1995 | BZ2_bzWrite |
| 1996 | if bzerror is BZ_OK |
| 1997 | (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless) |
| 1998 | BZ2_bzWriteClose |
| 1999 | otherwise |
| 2000 | </programlisting> |
| 2001 | |
| 2002 | </sect2> |
| 2003 | |
| 2004 | |
| 2005 | <sect2 id="bzwrite" xreflabel="BZ2_bzWrite"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2006 | <title>BZ2_bzWrite</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2007 | |
| 2008 | <programlisting> |
| 2009 | void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len ); |
| 2010 | </programlisting> |
| 2011 | |
| 2012 | <para>Absorbs <computeroutput>len</computeroutput> bytes from the |
| 2013 | buffer <computeroutput>buf</computeroutput>, eventually to be |
| 2014 | compressed and written to the file.</para> |
| 2015 | |
| 2016 | <para>Possible assignments to |
| 2017 | <computeroutput>bzerror</computeroutput>:</para> |
| 2018 | |
| 2019 | <programlisting> |
| 2020 | BZ_PARAM_ERROR |
| 2021 | if b is NULL or buf is NULL or len < 0 |
| 2022 | BZ_SEQUENCE_ERROR |
| 2023 | if b was opened with BZ2_bzReadOpen |
| 2024 | BZ_IO_ERROR |
| 2025 | if there is an error writing the compressed file. |
| 2026 | BZ_OK |
| 2027 | otherwise |
| 2028 | </programlisting> |
| 2029 | |
| 2030 | </sect2> |
| 2031 | |
| 2032 | |
| 2033 | <sect2 id="bzwriteclose" xreflabel="BZ2_bzWriteClose"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2034 | <title>BZ2_bzWriteClose</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2035 | |
| 2036 | <programlisting> |
| 2037 | void BZ2_bzWriteClose( int *bzerror, BZFILE* f, |
| 2038 | int abandon, |
| 2039 | unsigned int* nbytes_in, |
| 2040 | unsigned int* nbytes_out ); |
| 2041 | |
| 2042 | void BZ2_bzWriteClose64( int *bzerror, BZFILE* f, |
| 2043 | int abandon, |
| 2044 | unsigned int* nbytes_in_lo32, |
| 2045 | unsigned int* nbytes_in_hi32, |
| 2046 | unsigned int* nbytes_out_lo32, |
| 2047 | unsigned int* nbytes_out_hi32 ); |
| 2048 | </programlisting> |
| 2049 | |
| 2050 | <para>Compresses and flushes to the compressed file all data so |
| 2051 | far supplied by <computeroutput>BZ2_bzWrite</computeroutput>. |
| 2052 | The logical end-of-stream markers are also written, so subsequent |
| 2053 | calls to <computeroutput>BZ2_bzWrite</computeroutput> are |
| 2054 | illegal. All memory associated with the compressed file |
| 2055 | <computeroutput>b</computeroutput> is released. |
| 2056 | <computeroutput>fflush</computeroutput> is called on the |
| 2057 | compressed file, but it is not |
| 2058 | <computeroutput>fclose</computeroutput>'d.</para> |
| 2059 | |
| 2060 | <para>If <computeroutput>BZ2_bzWriteClose</computeroutput> is |
| 2061 | called to clean up after an error, the only action is to release |
| 2062 | the memory. The library records the error codes issued by |
| 2063 | previous calls, so this situation will be detected automatically. |
| 2064 | There is no attempt to complete the compression operation, nor to |
| 2065 | <computeroutput>fflush</computeroutput> the compressed file. You |
| 2066 | can force this behaviour to happen even in the case of no error, |
| 2067 | by passing a nonzero value to |
| 2068 | <computeroutput>abandon</computeroutput>.</para> |
| 2069 | |
| 2070 | <para>If <computeroutput>nbytes_in</computeroutput> is non-null, |
| 2071 | <computeroutput>*nbytes_in</computeroutput> will be set to be the |
| 2072 | total volume of uncompressed data handled. Similarly, |
| 2073 | <computeroutput>nbytes_out</computeroutput> will be set to the |
| 2074 | total volume of compressed data written. For compatibility with |
| 2075 | older versions of the library, |
| 2076 | <computeroutput>BZ2_bzWriteClose</computeroutput> only yields the |
| 2077 | lower 32 bits of these counts. Use |
| 2078 | <computeroutput>BZ2_bzWriteClose64</computeroutput> if you want |
| 2079 | the full 64 bit counts. These two functions are otherwise |
| 2080 | absolutely identical.</para> |
| 2081 | |
| 2082 | <para>Possible assignments to |
| 2083 | <computeroutput>bzerror</computeroutput>:</para> |
| 2084 | |
| 2085 | <programlisting> |
| 2086 | BZ_SEQUENCE_ERROR |
| 2087 | if b was opened with BZ2_bzReadOpen |
| 2088 | BZ_IO_ERROR |
| 2089 | if there is an error writing the compressed file |
| 2090 | BZ_OK |
| 2091 | otherwise |
| 2092 | </programlisting> |
| 2093 | |
| 2094 | </sect2> |
| 2095 | |
| 2096 | |
| 2097 | <sect2 id="embed" xreflabel="Handling embedded compressed data streams"> |
| 2098 | <title>Handling embedded compressed data streams</title> |
| 2099 | |
| 2100 | <para>The high-level library facilitates use of |
| 2101 | <computeroutput>bzip2</computeroutput> data streams which form |
| 2102 | some part of a surrounding, larger data stream.</para> |
| 2103 | |
| 2104 | <itemizedlist mark='bullet'> |
| 2105 | |
| 2106 | <listitem><para>For writing, the library takes an open file handle, |
| 2107 | writes compressed data to it, |
| 2108 | <computeroutput>fflush</computeroutput>es it but does not |
| 2109 | <computeroutput>fclose</computeroutput> it. The calling |
| 2110 | application can write its own data before and after the |
| 2111 | compressed data stream, using that same file handle.</para></listitem> |
| 2112 | |
| 2113 | <listitem><para>Reading is more complex, and the facilities are not as |
| 2114 | general as they could be since generality is hard to reconcile |
| 2115 | with efficiency. <computeroutput>BZ2_bzRead</computeroutput> |
| 2116 | reads from the compressed file in blocks of size |
| 2117 | <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes, and in |
| 2118 | doing so probably will overshoot the logical end of compressed |
| 2119 | stream. To recover this data once decompression has ended, |
| 2120 | call <computeroutput>BZ2_bzReadGetUnused</computeroutput> after |
| 2121 | the last call of <computeroutput>BZ2_bzRead</computeroutput> |
| 2122 | (the one returning |
| 2123 | <computeroutput>BZ_STREAM_END</computeroutput>) but before |
| 2124 | calling |
| 2125 | <computeroutput>BZ2_bzReadClose</computeroutput>.</para></listitem> |
| 2126 | |
| 2127 | </itemizedlist> |
| 2128 | |
| 2129 | <para>This mechanism makes it easy to decompress multiple |
| 2130 | <computeroutput>bzip2</computeroutput> streams placed end-to-end. |
| 2131 | As the end of one stream, when |
| 2132 | <computeroutput>BZ2_bzRead</computeroutput> returns |
| 2133 | <computeroutput>BZ_STREAM_END</computeroutput>, call |
| 2134 | <computeroutput>BZ2_bzReadGetUnused</computeroutput> to collect |
| 2135 | the unused data (copy it into your own buffer somewhere). That |
| 2136 | data forms the start of the next compressed stream. To start |
| 2137 | uncompressing that next stream, call |
| 2138 | <computeroutput>BZ2_bzReadOpen</computeroutput> again, feeding in |
| 2139 | the unused data via the <computeroutput>unused</computeroutput> / |
| 2140 | <computeroutput>nUnused</computeroutput> parameters. Keep doing |
| 2141 | this until <computeroutput>BZ_STREAM_END</computeroutput> return |
| 2142 | coincides with the physical end of file |
| 2143 | (<computeroutput>feof(f)</computeroutput>). In this situation |
| 2144 | <computeroutput>BZ2_bzReadGetUnused</computeroutput> will of |
| 2145 | course return no data.</para> |
| 2146 | |
| 2147 | <para>This should give some feel for how the high-level interface |
| 2148 | can be used. If you require extra flexibility, you'll have to |
| 2149 | bite the bullet and get to grips with the low-level |
| 2150 | interface.</para> |
| 2151 | |
| 2152 | </sect2> |
| 2153 | |
| 2154 | |
| 2155 | <sect2 id="std-rdwr" xreflabel="Standard file-reading/writing code"> |
| 2156 | <title>Standard file-reading/writing code</title> |
| 2157 | |
| 2158 | <para>Here's how you'd write data to a compressed file:</para> |
| 2159 | |
| 2160 | <programlisting> |
| 2161 | FILE* f; |
| 2162 | BZFILE* b; |
| 2163 | int nBuf; |
| 2164 | char buf[ /* whatever size you like */ ]; |
| 2165 | int bzerror; |
| 2166 | int nWritten; |
| 2167 | |
| 2168 | f = fopen ( "myfile.bz2", "w" ); |
| 2169 | if ( !f ) { |
| 2170 | /* handle error */ |
| 2171 | } |
| 2172 | b = BZ2_bzWriteOpen( &bzerror, f, 9 ); |
| 2173 | if (bzerror != BZ_OK) { |
| 2174 | BZ2_bzWriteClose ( b ); |
| 2175 | /* handle error */ |
| 2176 | } |
| 2177 | |
| 2178 | while ( /* condition */ ) { |
| 2179 | /* get data to write into buf, and set nBuf appropriately */ |
| 2180 | nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf ); |
| 2181 | if (bzerror == BZ_IO_ERROR) { |
| 2182 | BZ2_bzWriteClose ( &bzerror, b ); |
| 2183 | /* handle error */ |
| 2184 | } |
| 2185 | } |
| 2186 | |
| 2187 | BZ2_bzWriteClose( &bzerror, b ); |
| 2188 | if (bzerror == BZ_IO_ERROR) { |
| 2189 | /* handle error */ |
| 2190 | } |
| 2191 | </programlisting> |
| 2192 | |
| 2193 | <para>And to read from a compressed file:</para> |
| 2194 | |
| 2195 | <programlisting> |
| 2196 | FILE* f; |
| 2197 | BZFILE* b; |
| 2198 | int nBuf; |
| 2199 | char buf[ /* whatever size you like */ ]; |
| 2200 | int bzerror; |
| 2201 | int nWritten; |
| 2202 | |
| 2203 | f = fopen ( "myfile.bz2", "r" ); |
| 2204 | if ( !f ) { |
| 2205 | /* handle error */ |
| 2206 | } |
| 2207 | b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 ); |
| 2208 | if ( bzerror != BZ_OK ) { |
| 2209 | BZ2_bzReadClose ( &bzerror, b ); |
| 2210 | /* handle error */ |
| 2211 | } |
| 2212 | |
| 2213 | bzerror = BZ_OK; |
| 2214 | while ( bzerror == BZ_OK && /* arbitrary other conditions */) { |
| 2215 | nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ ); |
| 2216 | if ( bzerror == BZ_OK ) { |
| 2217 | /* do something with buf[0 .. nBuf-1] */ |
| 2218 | } |
| 2219 | } |
| 2220 | if ( bzerror != BZ_STREAM_END ) { |
| 2221 | BZ2_bzReadClose ( &bzerror, b ); |
| 2222 | /* handle error */ |
| 2223 | } else { |
| 2224 | BZ2_bzReadClose ( &bzerror, b ); |
| 2225 | } |
| 2226 | </programlisting> |
| 2227 | |
| 2228 | </sect2> |
| 2229 | |
| 2230 | </sect1> |
| 2231 | |
| 2232 | |
| 2233 | <sect1 id="util-fns" xreflabel="Utility functions"> |
| 2234 | <title>Utility functions</title> |
| 2235 | |
| 2236 | |
| 2237 | <sect2 id="bzbufftobuffcompress" xreflabel="BZ2_bzBuffToBuffCompress"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2238 | <title>BZ2_bzBuffToBuffCompress</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2239 | |
| 2240 | <programlisting> |
| 2241 | int BZ2_bzBuffToBuffCompress( char* dest, |
| 2242 | unsigned int* destLen, |
| 2243 | char* source, |
| 2244 | unsigned int sourceLen, |
| 2245 | int blockSize100k, |
| 2246 | int verbosity, |
| 2247 | int workFactor ); |
| 2248 | </programlisting> |
| 2249 | |
| 2250 | <para>Attempts to compress the data in <computeroutput>source[0 |
| 2251 | .. sourceLen-1]</computeroutput> into the destination buffer, |
| 2252 | <computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the |
| 2253 | destination buffer is big enough, |
| 2254 | <computeroutput>*destLen</computeroutput> is set to the size of |
| 2255 | the compressed data, and <computeroutput>BZ_OK</computeroutput> |
| 2256 | is returned. If the compressed data won't fit, |
| 2257 | <computeroutput>*destLen</computeroutput> is unchanged, and |
| 2258 | <computeroutput>BZ_OUTBUFF_FULL</computeroutput> is |
| 2259 | returned.</para> |
| 2260 | |
| 2261 | <para>Compression in this manner is a one-shot event, done with a |
| 2262 | single call to this function. The resulting compressed data is a |
| 2263 | complete <computeroutput>bzip2</computeroutput> format data |
| 2264 | stream. There is no mechanism for making additional calls to |
| 2265 | provide extra input data. If you want that kind of mechanism, |
| 2266 | use the low-level interface.</para> |
| 2267 | |
| 2268 | <para>For the meaning of parameters |
| 2269 | <computeroutput>blockSize100k</computeroutput>, |
| 2270 | <computeroutput>verbosity</computeroutput> and |
| 2271 | <computeroutput>workFactor</computeroutput>, see |
| 2272 | <computeroutput>BZ2_bzCompressInit</computeroutput>.</para> |
| 2273 | |
| 2274 | <para>To guarantee that the compressed data will fit in its |
| 2275 | buffer, allocate an output buffer of size 1% larger than the |
| 2276 | uncompressed data, plus six hundred extra bytes.</para> |
| 2277 | |
| 2278 | <para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> |
| 2279 | will not write data at or beyond |
| 2280 | <computeroutput>dest[*destLen]</computeroutput>, even in case of |
| 2281 | buffer overflow.</para> |
| 2282 | |
| 2283 | <para>Possible return values:</para> |
| 2284 | |
| 2285 | <programlisting> |
| 2286 | BZ_CONFIG_ERROR |
| 2287 | if the library has been mis-compiled |
| 2288 | BZ_PARAM_ERROR |
| 2289 | if dest is NULL or destLen is NULL |
| 2290 | or blockSize100k < 1 or blockSize100k > 9 |
| 2291 | or verbosity < 0 or verbosity > 4 |
| 2292 | or workFactor < 0 or workFactor > 250 |
| 2293 | BZ_MEM_ERROR |
| 2294 | if insufficient memory is available |
| 2295 | BZ_OUTBUFF_FULL |
| 2296 | if the size of the compressed data exceeds *destLen |
| 2297 | BZ_OK |
| 2298 | otherwise |
| 2299 | </programlisting> |
| 2300 | |
| 2301 | </sect2> |
| 2302 | |
| 2303 | |
| 2304 | <sect2 id="bzbufftobuffdecompress" xreflabel="BZ2_bzBuffToBuffDecompress"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2305 | <title>BZ2_bzBuffToBuffDecompress</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2306 | |
| 2307 | <programlisting> |
| 2308 | int BZ2_bzBuffToBuffDecompress( char* dest, |
| 2309 | unsigned int* destLen, |
| 2310 | char* source, |
| 2311 | unsigned int sourceLen, |
| 2312 | int small, |
| 2313 | int verbosity ); |
| 2314 | </programlisting> |
| 2315 | |
| 2316 | <para>Attempts to decompress the data in <computeroutput>source[0 |
| 2317 | .. sourceLen-1]</computeroutput> into the destination buffer, |
| 2318 | <computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the |
| 2319 | destination buffer is big enough, |
| 2320 | <computeroutput>*destLen</computeroutput> is set to the size of |
| 2321 | the uncompressed data, and <computeroutput>BZ_OK</computeroutput> |
| 2322 | is returned. If the compressed data won't fit, |
| 2323 | <computeroutput>*destLen</computeroutput> is unchanged, and |
| 2324 | <computeroutput>BZ_OUTBUFF_FULL</computeroutput> is |
| 2325 | returned.</para> |
| 2326 | |
| 2327 | <para><computeroutput>source</computeroutput> is assumed to hold |
| 2328 | a complete <computeroutput>bzip2</computeroutput> format data |
| 2329 | stream. |
| 2330 | <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> tries |
| 2331 | to decompress the entirety of the stream into the output |
| 2332 | buffer.</para> |
| 2333 | |
| 2334 | <para>For the meaning of parameters |
| 2335 | <computeroutput>small</computeroutput> and |
| 2336 | <computeroutput>verbosity</computeroutput>, see |
| 2337 | <computeroutput>BZ2_bzDecompressInit</computeroutput>.</para> |
| 2338 | |
| 2339 | <para>Because the compression ratio of the compressed data cannot |
| 2340 | be known in advance, there is no easy way to guarantee that the |
| 2341 | output buffer will be big enough. You may of course make |
| 2342 | arrangements in your code to record the size of the uncompressed |
| 2343 | data, but such a mechanism is beyond the scope of this |
| 2344 | library.</para> |
| 2345 | |
| 2346 | <para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> |
| 2347 | will not write data at or beyond |
| 2348 | <computeroutput>dest[*destLen]</computeroutput>, even in case of |
| 2349 | buffer overflow.</para> |
| 2350 | |
| 2351 | <para>Possible return values:</para> |
| 2352 | |
| 2353 | <programlisting> |
| 2354 | BZ_CONFIG_ERROR |
| 2355 | if the library has been mis-compiled |
| 2356 | BZ_PARAM_ERROR |
| 2357 | if dest is NULL or destLen is NULL |
| 2358 | or small != 0 && small != 1 |
| 2359 | or verbosity < 0 or verbosity > 4 |
| 2360 | BZ_MEM_ERROR |
| 2361 | if insufficient memory is available |
| 2362 | BZ_OUTBUFF_FULL |
| 2363 | if the size of the compressed data exceeds *destLen |
| 2364 | BZ_DATA_ERROR |
| 2365 | if a data integrity error was detected in the compressed data |
| 2366 | BZ_DATA_ERROR_MAGIC |
| 2367 | if the compressed data doesn't begin with the right magic bytes |
| 2368 | BZ_UNEXPECTED_EOF |
| 2369 | if the compressed data ends unexpectedly |
| 2370 | BZ_OK |
| 2371 | otherwise |
| 2372 | </programlisting> |
| 2373 | |
| 2374 | </sect2> |
| 2375 | |
| 2376 | </sect1> |
| 2377 | |
| 2378 | |
| 2379 | <sect1 id="zlib-compat" xreflabel="zlib compatibility functions"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2380 | <title>zlib compatibility functions</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2381 | |
| 2382 | <para>Yoshioka Tsuneo has contributed some functions to give |
| 2383 | better <computeroutput>zlib</computeroutput> compatibility. |
| 2384 | These functions are <computeroutput>BZ2_bzopen</computeroutput>, |
| 2385 | <computeroutput>BZ2_bzread</computeroutput>, |
| 2386 | <computeroutput>BZ2_bzwrite</computeroutput>, |
| 2387 | <computeroutput>BZ2_bzflush</computeroutput>, |
| 2388 | <computeroutput>BZ2_bzclose</computeroutput>, |
| 2389 | <computeroutput>BZ2_bzerror</computeroutput> and |
| 2390 | <computeroutput>BZ2_bzlibVersion</computeroutput>. These |
| 2391 | functions are not (yet) officially part of the library. If they |
| 2392 | break, you get to keep all the pieces. Nevertheless, I think |
| 2393 | they work ok.</para> |
| 2394 | |
| 2395 | <programlisting> |
| 2396 | typedef void BZFILE; |
| 2397 | |
| 2398 | const char * BZ2_bzlibVersion ( void ); |
| 2399 | </programlisting> |
| 2400 | |
| 2401 | <para>Returns a string indicating the library version.</para> |
| 2402 | |
| 2403 | <programlisting> |
| 2404 | BZFILE * BZ2_bzopen ( const char *path, const char *mode ); |
| 2405 | BZFILE * BZ2_bzdopen ( int fd, const char *mode ); |
| 2406 | </programlisting> |
| 2407 | |
| 2408 | <para>Opens a <computeroutput>.bz2</computeroutput> file for |
| 2409 | reading or writing, using either its name or a pre-existing file |
| 2410 | descriptor. Analogous to <computeroutput>fopen</computeroutput> |
| 2411 | and <computeroutput>fdopen</computeroutput>.</para> |
| 2412 | |
| 2413 | <programlisting> |
| 2414 | int BZ2_bzread ( BZFILE* b, void* buf, int len ); |
| 2415 | int BZ2_bzwrite ( BZFILE* b, void* buf, int len ); |
| 2416 | </programlisting> |
| 2417 | |
| 2418 | <para>Reads/writes data from/to a previously opened |
| 2419 | <computeroutput>BZFILE</computeroutput>. Analogous to |
| 2420 | <computeroutput>fread</computeroutput> and |
| 2421 | <computeroutput>fwrite</computeroutput>.</para> |
| 2422 | |
| 2423 | <programlisting> |
| 2424 | int BZ2_bzflush ( BZFILE* b ); |
| 2425 | void BZ2_bzclose ( BZFILE* b ); |
| 2426 | </programlisting> |
| 2427 | |
| 2428 | <para>Flushes/closes a <computeroutput>BZFILE</computeroutput>. |
| 2429 | <computeroutput>BZ2_bzflush</computeroutput> doesn't actually do |
| 2430 | anything. Analogous to <computeroutput>fflush</computeroutput> |
| 2431 | and <computeroutput>fclose</computeroutput>.</para> |
| 2432 | |
| 2433 | <programlisting> |
| 2434 | const char * BZ2_bzerror ( BZFILE *b, int *errnum ) |
| 2435 | </programlisting> |
| 2436 | |
| 2437 | <para>Returns a string describing the more recent error status of |
| 2438 | <computeroutput>b</computeroutput>, and also sets |
| 2439 | <computeroutput>*errnum</computeroutput> to its numerical |
| 2440 | value.</para> |
| 2441 | |
| 2442 | </sect1> |
| 2443 | |
| 2444 | |
| 2445 | <sect1 id="stdio-free" |
| 2446 | xreflabel="Using the library in a stdio-free environment"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2447 | <title>Using the library in a stdio-free environment</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2448 | |
| 2449 | |
| 2450 | <sect2 id="stdio-bye" xreflabel="Getting rid of stdio"> |
Nick Kralevich | 172b266 | 2010-09-20 17:21:30 -0700 | [diff] [blame] | 2451 | <title>Getting rid of stdio</title> |
The Android Open Source Project | cfb3b27 | 2009-03-03 19:29:20 -0800 | [diff] [blame] | 2452 | |
| 2453 | <para>In a deeply embedded application, you might want to use |
| 2454 | just the memory-to-memory functions. You can do this |
| 2455 | conveniently by compiling the library with preprocessor symbol |
| 2456 | <computeroutput>BZ_NO_STDIO</computeroutput> defined. Doing this |
| 2457 | gives you a library containing only the following eight |
| 2458 | functions:</para> |
| 2459 | |
| 2460 | <para><computeroutput>BZ2_bzCompressInit</computeroutput>, |
| 2461 | <computeroutput>BZ2_bzCompress</computeroutput>, |
| 2462 | <computeroutput>BZ2_bzCompressEnd</computeroutput> |
| 2463 | <computeroutput>BZ2_bzDecompressInit</computeroutput>, |
| 2464 | <computeroutput>BZ2_bzDecompress</computeroutput>, |
| 2465 | <computeroutput>BZ2_bzDecompressEnd</computeroutput> |
| 2466 | <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput>, |
| 2467 | <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput></para> |
| 2468 | |
| 2469 | <para>When compiled like this, all functions will ignore |
| 2470 | <computeroutput>verbosity</computeroutput> settings.</para> |
| 2471 | |
| 2472 | </sect2> |
| 2473 | |
| 2474 | |
| 2475 | <sect2 id="critical-error" xreflabel="Critical error handling"> |
| 2476 | <title>Critical error handling</title> |
| 2477 | |
| 2478 | <para><computeroutput>libbzip2</computeroutput> contains a number |
| 2479 | of internal assertion checks which should, needless to say, never |
| 2480 | be activated. Nevertheless, if an assertion should fail, |
| 2481 | behaviour depends on whether or not the library was compiled with |
| 2482 | <computeroutput>BZ_NO_STDIO</computeroutput> set.</para> |
| 2483 | |
| 2484 | <para>For a normal compile, an assertion failure yields the |
| 2485 | message:</para> |
| 2486 | |
| 2487 | <blockquote> |
| 2488 | <para>bzip2/libbzip2: internal error number N.</para> |
| 2489 | <para>This is a bug in bzip2/libbzip2, &bz-version; of &bz-date;. |
| 2490 | Please report it to me at: &bz-email;. If this happened |
| 2491 | when you were using some program which uses libbzip2 as a |
| 2492 | component, you should also report this bug to the author(s) |
| 2493 | of that program. Please make an effort to report this bug; |
| 2494 | timely and accurate bug reports eventually lead to higher |
| 2495 | quality software. Thanks. Julian Seward, &bz-date;. |
| 2496 | </para></blockquote> |
| 2497 | |
| 2498 | <para>where <computeroutput>N</computeroutput> is some error code |
| 2499 | number. If <computeroutput>N == 1007</computeroutput>, it also |
| 2500 | prints some extra text advising the reader that unreliable memory |
| 2501 | is often associated with internal error 1007. (This is a |
| 2502 | frequently-observed-phenomenon with versions 1.0.0/1.0.1).</para> |
| 2503 | |
| 2504 | <para><computeroutput>exit(3)</computeroutput> is then |
| 2505 | called.</para> |
| 2506 | |
| 2507 | <para>For a <computeroutput>stdio</computeroutput>-free library, |
| 2508 | assertion failures result in a call to a function declared |
| 2509 | as:</para> |
| 2510 | |
| 2511 | <programlisting> |
| 2512 | extern void bz_internal_error ( int errcode ); |
| 2513 | </programlisting> |
| 2514 | |
| 2515 | <para>The relevant code is passed as a parameter. You should |
| 2516 | supply such a function.</para> |
| 2517 | |
| 2518 | <para>In either case, once an assertion failure has occurred, any |
| 2519 | <computeroutput>bz_stream</computeroutput> records involved can |
| 2520 | be regarded as invalid. You should not attempt to resume normal |
| 2521 | operation with them.</para> |
| 2522 | |
| 2523 | <para>You may, of course, change critical error handling to suit |
| 2524 | your needs. As I said above, critical errors indicate bugs in |
| 2525 | the library and should not occur. All "normal" error situations |
| 2526 | are indicated via error return codes from functions, and can be |
| 2527 | recovered from.</para> |
| 2528 | |
| 2529 | </sect2> |
| 2530 | |
| 2531 | </sect1> |
| 2532 | |
| 2533 | |
| 2534 | <sect1 id="win-dll" xreflabel="Making a Windows DLL"> |
| 2535 | <title>Making a Windows DLL</title> |
| 2536 | |
| 2537 | <para>Everything related to Windows has been contributed by |
| 2538 | Yoshioka Tsuneo |
| 2539 | (<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>), so |
| 2540 | you should send your queries to him (but perhaps Cc: me, |
| 2541 | <computeroutput>&bz-email;</computeroutput>).</para> |
| 2542 | |
| 2543 | <para>My vague understanding of what to do is: using Visual C++ |
| 2544 | 5.0, open the project file |
| 2545 | <computeroutput>libbz2.dsp</computeroutput>, and build. That's |
| 2546 | all.</para> |
| 2547 | |
| 2548 | <para>If you can't open the project file for some reason, make a |
| 2549 | new one, naming these files: |
| 2550 | <computeroutput>blocksort.c</computeroutput>, |
| 2551 | <computeroutput>bzlib.c</computeroutput>, |
| 2552 | <computeroutput>compress.c</computeroutput>, |
| 2553 | <computeroutput>crctable.c</computeroutput>, |
| 2554 | <computeroutput>decompress.c</computeroutput>, |
| 2555 | <computeroutput>huffman.c</computeroutput>, |
| 2556 | <computeroutput>randtable.c</computeroutput> and |
| 2557 | <computeroutput>libbz2.def</computeroutput>. You will also need |
| 2558 | to name the header files <computeroutput>bzlib.h</computeroutput> |
| 2559 | and <computeroutput>bzlib_private.h</computeroutput>.</para> |
| 2560 | |
| 2561 | <para>If you don't use VC++, you may need to define the |
| 2562 | proprocessor symbol |
| 2563 | <computeroutput>_WIN32</computeroutput>.</para> |
| 2564 | |
| 2565 | <para>Finally, <computeroutput>dlltest.c</computeroutput> is a |
| 2566 | sample program using the DLL. It has a project file, |
| 2567 | <computeroutput>dlltest.dsp</computeroutput>.</para> |
| 2568 | |
| 2569 | <para>If you just want a makefile for Visual C, have a look at |
| 2570 | <computeroutput>makefile.msc</computeroutput>.</para> |
| 2571 | |
| 2572 | <para>Be aware that if you compile |
| 2573 | <computeroutput>bzip2</computeroutput> itself on Win32, you must |
| 2574 | set <computeroutput>BZ_UNIX</computeroutput> to 0 and |
| 2575 | <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the file |
| 2576 | <computeroutput>bzip2.c</computeroutput>, before compiling. |
| 2577 | Otherwise the resulting binary won't work correctly.</para> |
| 2578 | |
| 2579 | <para>I haven't tried any of this stuff myself, but it all looks |
| 2580 | plausible.</para> |
| 2581 | |
| 2582 | </sect1> |
| 2583 | |
| 2584 | </chapter> |
| 2585 | |
| 2586 | |
| 2587 | |
| 2588 | <chapter id="misc" xreflabel="Miscellanea"> |
| 2589 | <title>Miscellanea</title> |
| 2590 | |
| 2591 | <para>These are just some random thoughts of mine. Your mileage |
| 2592 | may vary.</para> |
| 2593 | |
| 2594 | |
| 2595 | <sect1 id="limits" xreflabel="Limitations of the compressed file format"> |
| 2596 | <title>Limitations of the compressed file format</title> |
| 2597 | |
| 2598 | <para><computeroutput>bzip2-1.0.X</computeroutput>, |
| 2599 | <computeroutput>0.9.5</computeroutput> and |
| 2600 | <computeroutput>0.9.0</computeroutput> use exactly the same file |
| 2601 | format as the original version, |
| 2602 | <computeroutput>bzip2-0.1</computeroutput>. This decision was |
| 2603 | made in the interests of stability. Creating yet another |
| 2604 | incompatible compressed file format would create further |
| 2605 | confusion and disruption for users.</para> |
| 2606 | |
| 2607 | <para>Nevertheless, this is not a painless decision. Development |
| 2608 | work since the release of |
| 2609 | <computeroutput>bzip2-0.1</computeroutput> in August 1997 has |
| 2610 | shown complexities in the file format which slow down |
| 2611 | decompression and, in retrospect, are unnecessary. These |
| 2612 | are:</para> |
| 2613 | |
| 2614 | <itemizedlist mark='bullet'> |
| 2615 | |
| 2616 | <listitem><para>The run-length encoder, which is the first of the |
| 2617 | compression transformations, is entirely irrelevant. The |
| 2618 | original purpose was to protect the sorting algorithm from the |
| 2619 | very worst case input: a string of repeated symbols. But |
| 2620 | algorithm steps Q6a and Q6b in the original Burrows-Wheeler |
| 2621 | technical report (SRC-124) show how repeats can be handled |
| 2622 | without difficulty in block sorting.</para></listitem> |
| 2623 | |
| 2624 | <listitem><para>The randomisation mechanism doesn't really need to be |
| 2625 | there. Udi Manber and Gene Myers published a suffix array |
| 2626 | construction algorithm a few years back, which can be employed |
| 2627 | to sort any block, no matter how repetitive, in O(N log N) |
| 2628 | time. Subsequent work by Kunihiko Sadakane has produced a |
| 2629 | derivative O(N (log N)^2) algorithm which usually outperforms |
| 2630 | the Manber-Myers algorithm.</para> |
| 2631 | |
| 2632 | <para>I could have changed to Sadakane's algorithm, but I find |
| 2633 | it to be slower than <computeroutput>bzip2</computeroutput>'s |
| 2634 | existing algorithm for most inputs, and the randomisation |
| 2635 | mechanism protects adequately against bad cases. I didn't |
| 2636 | think it was a good tradeoff to make. Partly this is due to |
| 2637 | the fact that I was not flooded with email complaints about |
| 2638 | <computeroutput>bzip2-0.1</computeroutput>'s performance on |
| 2639 | repetitive data, so perhaps it isn't a problem for real |
| 2640 | inputs.</para> |
| 2641 | |
| 2642 | <para>Probably the best long-term solution, and the one I have |
| 2643 | incorporated into 0.9.5 and above, is to use the existing |
| 2644 | sorting algorithm initially, and fall back to a O(N (log N)^2) |
| 2645 | algorithm if the standard algorithm gets into |
| 2646 | difficulties.</para></listitem> |
| 2647 | |
| 2648 | <listitem><para>The compressed file format was never designed to be |
| 2649 | handled by a library, and I have had to jump though some hoops |
| 2650 | to produce an efficient implementation of decompression. It's |
| 2651 | a bit hairy. Try passing |
| 2652 | <computeroutput>decompress.c</computeroutput> through the C |
| 2653 | preprocessor and you'll see what I mean. Much of this |
| 2654 | complexity could have been avoided if the compressed size of |
| 2655 | each block of data was recorded in the data stream.</para></listitem> |
| 2656 | |
| 2657 | <listitem><para>An Adler-32 checksum, rather than a CRC32 checksum, |
| 2658 | would be faster to compute.</para></listitem> |
| 2659 | |
| 2660 | </itemizedlist> |
| 2661 | |
| 2662 | <para>It would be fair to say that the |
| 2663 | <computeroutput>bzip2</computeroutput> format was frozen before I |
| 2664 | properly and fully understood the performance consequences of |
| 2665 | doing so.</para> |
| 2666 | |
| 2667 | <para>Improvements which I was able to incorporate into 0.9.0, |
| 2668 | despite using the same file format, are:</para> |
| 2669 | |
| 2670 | <itemizedlist mark='bullet'> |
| 2671 | |
| 2672 | <listitem><para>Single array implementation of the inverse BWT. This |
| 2673 | significantly speeds up decompression, presumably because it |
| 2674 | reduces the number of cache misses.</para></listitem> |
| 2675 | |
| 2676 | <listitem><para>Faster inverse MTF transform for large MTF values. |
| 2677 | The new implementation is based on the notion of sliding blocks |
| 2678 | of values.</para></listitem> |
| 2679 | |
| 2680 | <listitem><para><computeroutput>bzip2-0.9.0</computeroutput> now reads |
| 2681 | and writes files with <computeroutput>fread</computeroutput> |
| 2682 | and <computeroutput>fwrite</computeroutput>; version 0.1 used |
| 2683 | <computeroutput>putc</computeroutput> and |
| 2684 | <computeroutput>getc</computeroutput>. Duh! Well, you live |
| 2685 | and learn.</para></listitem> |
| 2686 | |
| 2687 | </itemizedlist> |
| 2688 | |
| 2689 | <para>Further ahead, it would be nice to be able to do random |
| 2690 | access into files. This will require some careful design of |
| 2691 | compressed file formats.</para> |
| 2692 | |
| 2693 | </sect1> |
| 2694 | |
| 2695 | |
| 2696 | <sect1 id="port-issues" xreflabel="Portability issues"> |
| 2697 | <title>Portability issues</title> |
| 2698 | |
| 2699 | <para>After some consideration, I have decided not to use GNU |
| 2700 | <computeroutput>autoconf</computeroutput> to configure 0.9.5 or |
| 2701 | 1.0.</para> |
| 2702 | |
| 2703 | <para><computeroutput>autoconf</computeroutput>, admirable and |
| 2704 | wonderful though it is, mainly assists with portability problems |
| 2705 | between Unix-like platforms. But |
| 2706 | <computeroutput>bzip2</computeroutput> doesn't have much in the |
| 2707 | way of portability problems on Unix; most of the difficulties |
| 2708 | appear when porting to the Mac, or to Microsoft's operating |
| 2709 | systems. <computeroutput>autoconf</computeroutput> doesn't help |
| 2710 | in those cases, and brings in a whole load of new |
| 2711 | complexity.</para> |
| 2712 | |
| 2713 | <para>Most people should be able to compile the library and |
| 2714 | program under Unix straight out-of-the-box, so to speak, |
| 2715 | especially if you have a version of GNU C available.</para> |
| 2716 | |
| 2717 | <para>There are a couple of |
| 2718 | <computeroutput>__inline__</computeroutput> directives in the |
| 2719 | code. GNU C (<computeroutput>gcc</computeroutput>) should be |
| 2720 | able to handle them. If you're not using GNU C, your C compiler |
| 2721 | shouldn't see them at all. If your compiler does, for some |
| 2722 | reason, see them and doesn't like them, just |
| 2723 | <computeroutput>#define</computeroutput> |
| 2724 | <computeroutput>__inline__</computeroutput> to be |
| 2725 | <computeroutput>/* */</computeroutput>. One easy way to do this |
| 2726 | is to compile with the flag |
| 2727 | <computeroutput>-D__inline__=</computeroutput>, which should be |
| 2728 | understood by most Unix compilers.</para> |
| 2729 | |
| 2730 | <para>If you still have difficulties, try compiling with the |
| 2731 | macro <computeroutput>BZ_STRICT_ANSI</computeroutput> defined. |
| 2732 | This should enable you to build the library in a strictly ANSI |
| 2733 | compliant environment. Building the program itself like this is |
| 2734 | dangerous and not supported, since you remove |
| 2735 | <computeroutput>bzip2</computeroutput>'s checks against |
| 2736 | compressing directories, symbolic links, devices, and other |
| 2737 | not-really-a-file entities. This could cause filesystem |
| 2738 | corruption!</para> |
| 2739 | |
| 2740 | <para>One other thing: if you create a |
| 2741 | <computeroutput>bzip2</computeroutput> binary for public distribution, |
| 2742 | please consider linking it statically (<computeroutput>gcc |
| 2743 | -static</computeroutput>). This avoids all sorts of library-version |
| 2744 | issues that others may encounter later on.</para> |
| 2745 | |
| 2746 | <para>If you build <computeroutput>bzip2</computeroutput> on |
| 2747 | Win32, you must set <computeroutput>BZ_UNIX</computeroutput> to 0 |
| 2748 | and <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the |
| 2749 | file <computeroutput>bzip2.c</computeroutput>, before compiling. |
| 2750 | Otherwise the resulting binary won't work correctly.</para> |
| 2751 | |
| 2752 | </sect1> |
| 2753 | |
| 2754 | |
| 2755 | <sect1 id="bugs" xreflabel="Reporting bugs"> |
| 2756 | <title>Reporting bugs</title> |
| 2757 | |
| 2758 | <para>I tried pretty hard to make sure |
| 2759 | <computeroutput>bzip2</computeroutput> is bug free, both by |
| 2760 | design and by testing. Hopefully you'll never need to read this |
| 2761 | section for real.</para> |
| 2762 | |
| 2763 | <para>Nevertheless, if <computeroutput>bzip2</computeroutput> dies |
| 2764 | with a segmentation fault, a bus error or an internal assertion |
| 2765 | failure, it will ask you to email me a bug report. Experience from |
| 2766 | years of feedback of bzip2 users indicates that almost all these |
| 2767 | problems can be traced to either compiler bugs or hardware |
| 2768 | problems.</para> |
| 2769 | |
| 2770 | <itemizedlist mark='bullet'> |
| 2771 | |
| 2772 | <listitem><para>Recompile the program with no optimisation, and |
| 2773 | see if it works. And/or try a different compiler. I heard all |
| 2774 | sorts of stories about various flavours of GNU C (and other |
| 2775 | compilers) generating bad code for |
| 2776 | <computeroutput>bzip2</computeroutput>, and I've run across two |
| 2777 | such examples myself.</para> |
| 2778 | |
| 2779 | <para>2.7.X versions of GNU C are known to generate bad code |
| 2780 | from time to time, at high optimisation levels. If you get |
| 2781 | problems, try using the flags |
| 2782 | <computeroutput>-O2</computeroutput> |
| 2783 | <computeroutput>-fomit-frame-pointer</computeroutput> |
| 2784 | <computeroutput>-fno-strength-reduce</computeroutput>. You |
| 2785 | should specifically <emphasis>not</emphasis> use |
| 2786 | <computeroutput>-funroll-loops</computeroutput>.</para> |
| 2787 | |
| 2788 | <para>You may notice that the Makefile runs six tests as part |
| 2789 | of the build process. If the program passes all of these, it's |
| 2790 | a pretty good (but not 100%) indication that the compiler has |
| 2791 | done its job correctly.</para></listitem> |
| 2792 | |
| 2793 | <listitem><para>If <computeroutput>bzip2</computeroutput> |
| 2794 | crashes randomly, and the crashes are not repeatable, you may |
| 2795 | have a flaky memory subsystem. |
| 2796 | <computeroutput>bzip2</computeroutput> really hammers your |
| 2797 | memory hierarchy, and if it's a bit marginal, you may get these |
| 2798 | problems. Ditto if your disk or I/O subsystem is slowly |
| 2799 | failing. Yup, this really does happen.</para> |
| 2800 | |
| 2801 | <para>Try using a different machine of the same type, and see |
| 2802 | if you can repeat the problem.</para></listitem> |
| 2803 | |
| 2804 | <listitem><para>This isn't really a bug, but ... If |
| 2805 | <computeroutput>bzip2</computeroutput> tells you your file is |
| 2806 | corrupted on decompression, and you obtained the file via FTP, |
| 2807 | there is a possibility that you forgot to tell FTP to do a |
| 2808 | binary mode transfer. That absolutely will cause the file to |
| 2809 | be non-decompressible. You'll have to transfer it |
| 2810 | again.</para></listitem> |
| 2811 | |
| 2812 | </itemizedlist> |
| 2813 | |
| 2814 | <para>If you've incorporated |
| 2815 | <computeroutput>libbzip2</computeroutput> into your own program |
| 2816 | and are getting problems, please, please, please, check that the |
| 2817 | parameters you are passing in calls to the library, are correct, |
| 2818 | and in accordance with what the documentation says is allowable. |
| 2819 | I have tried to make the library robust against such problems, |
| 2820 | but I'm sure I haven't succeeded.</para> |
| 2821 | |
| 2822 | <para>Finally, if the above comments don't help, you'll have to |
| 2823 | send me a bug report. Now, it's just amazing how many people |
| 2824 | will send me a bug report saying something like:</para> |
| 2825 | |
| 2826 | <programlisting> |
| 2827 | bzip2 crashed with segmentation fault on my machine |
| 2828 | </programlisting> |
| 2829 | |
| 2830 | <para>and absolutely nothing else. Needless to say, a such a |
| 2831 | report is <emphasis>totally, utterly, completely and |
| 2832 | comprehensively 100% useless; a waste of your time, my time, and |
| 2833 | net bandwidth</emphasis>. With no details at all, there's no way |
| 2834 | I can possibly begin to figure out what the problem is.</para> |
| 2835 | |
| 2836 | <para>The rules of the game are: facts, facts, facts. Don't omit |
| 2837 | them because "oh, they won't be relevant". At the bare |
| 2838 | minimum:</para> |
| 2839 | |
| 2840 | <programlisting> |
| 2841 | Machine type. Operating system version. |
| 2842 | Exact version of bzip2 (do bzip2 -V). |
| 2843 | Exact version of the compiler used. |
| 2844 | Flags passed to the compiler. |
| 2845 | </programlisting> |
| 2846 | |
| 2847 | <para>However, the most important single thing that will help me |
| 2848 | is the file that you were trying to compress or decompress at the |
| 2849 | time the problem happened. Without that, my ability to do |
| 2850 | anything more than speculate about the cause, is limited.</para> |
| 2851 | |
| 2852 | </sect1> |
| 2853 | |
| 2854 | |
| 2855 | <sect1 id="package" xreflabel="Did you get the right package?"> |
| 2856 | <title>Did you get the right package?</title> |
| 2857 | |
| 2858 | <para><computeroutput>bzip2</computeroutput> is a resource hog. |
| 2859 | It soaks up large amounts of CPU cycles and memory. Also, it |
| 2860 | gives very large latencies. In the worst case, you can feed many |
| 2861 | megabytes of uncompressed data into the library before getting |
| 2862 | any compressed output, so this probably rules out applications |
| 2863 | requiring interactive behaviour.</para> |
| 2864 | |
| 2865 | <para>These aren't faults of my implementation, I hope, but more |
| 2866 | an intrinsic property of the Burrows-Wheeler transform |
| 2867 | (unfortunately). Maybe this isn't what you want.</para> |
| 2868 | |
| 2869 | <para>If you want a compressor and/or library which is faster, |
| 2870 | uses less memory but gets pretty good compression, and has |
| 2871 | minimal latency, consider Jean-loup Gailly's and Mark Adler's |
| 2872 | work, <computeroutput>zlib-1.2.1</computeroutput> and |
| 2873 | <computeroutput>gzip-1.2.4</computeroutput>. Look for them at |
| 2874 | <ulink url="http://www.zlib.org">http://www.zlib.org</ulink> and |
| 2875 | <ulink url="http://www.gzip.org">http://www.gzip.org</ulink> |
| 2876 | respectively.</para> |
| 2877 | |
| 2878 | <para>For something faster and lighter still, you might try Markus F |
| 2879 | X J Oberhumer's <computeroutput>LZO</computeroutput> real-time |
| 2880 | compression/decompression library, at |
| 2881 | <ulink url="http://www.oberhumer.com/opensource">http://www.oberhumer.com/opensource</ulink>.</para> |
| 2882 | |
| 2883 | </sect1> |
| 2884 | |
| 2885 | |
| 2886 | |
| 2887 | <sect1 id="reading" xreflabel="Further Reading"> |
| 2888 | <title>Further Reading</title> |
| 2889 | |
| 2890 | <para><computeroutput>bzip2</computeroutput> is not research |
| 2891 | work, in the sense that it doesn't present any new ideas. |
| 2892 | Rather, it's an engineering exercise based on existing |
| 2893 | ideas.</para> |
| 2894 | |
| 2895 | <para>Four documents describe essentially all the ideas behind |
| 2896 | <computeroutput>bzip2</computeroutput>:</para> |
| 2897 | |
| 2898 | <literallayout>Michael Burrows and D. J. Wheeler: |
| 2899 | "A block-sorting lossless data compression algorithm" |
| 2900 | 10th May 1994. |
| 2901 | Digital SRC Research Report 124. |
| 2902 | ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz |
| 2903 | If you have trouble finding it, try searching at the |
| 2904 | New Zealand Digital Library, http://www.nzdl.org. |
| 2905 | |
| 2906 | Daniel S. Hirschberg and Debra A. LeLewer |
| 2907 | "Efficient Decoding of Prefix Codes" |
| 2908 | Communications of the ACM, April 1990, Vol 33, Number 4. |
| 2909 | You might be able to get an electronic copy of this |
| 2910 | from the ACM Digital Library. |
| 2911 | |
| 2912 | David J. Wheeler |
| 2913 | Program bred3.c and accompanying document bred3.ps. |
| 2914 | This contains the idea behind the multi-table Huffman coding scheme. |
| 2915 | ftp://ftp.cl.cam.ac.uk/users/djw3/ |
| 2916 | |
| 2917 | Jon L. Bentley and Robert Sedgewick |
| 2918 | "Fast Algorithms for Sorting and Searching Strings" |
| 2919 | Available from Sedgewick's web page, |
| 2920 | www.cs.princeton.edu/~rs |
| 2921 | </literallayout> |
| 2922 | |
| 2923 | <para>The following paper gives valuable additional insights into |
| 2924 | the algorithm, but is not immediately the basis of any code used |
| 2925 | in bzip2.</para> |
| 2926 | |
| 2927 | <literallayout>Peter Fenwick: |
| 2928 | Block Sorting Text Compression |
| 2929 | Proceedings of the 19th Australasian Computer Science Conference, |
| 2930 | Melbourne, Australia. Jan 31 - Feb 2, 1996. |
| 2931 | ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</literallayout> |
| 2932 | |
| 2933 | <para>Kunihiko Sadakane's sorting algorithm, mentioned above, is |
| 2934 | available from:</para> |
| 2935 | |
| 2936 | <literallayout>http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz |
| 2937 | </literallayout> |
| 2938 | |
| 2939 | <para>The Manber-Myers suffix array construction algorithm is |
| 2940 | described in a paper available from:</para> |
| 2941 | |
| 2942 | <literallayout>http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps |
| 2943 | </literallayout> |
| 2944 | |
| 2945 | <para>Finally, the following papers document some |
| 2946 | investigations I made into the performance of sorting |
| 2947 | and decompression algorithms:</para> |
| 2948 | |
| 2949 | <literallayout>Julian Seward |
| 2950 | On the Performance of BWT Sorting Algorithms |
| 2951 | Proceedings of the IEEE Data Compression Conference 2000 |
| 2952 | Snowbird, Utah. 28-30 March 2000. |
| 2953 | |
| 2954 | Julian Seward |
| 2955 | Space-time Tradeoffs in the Inverse B-W Transform |
| 2956 | Proceedings of the IEEE Data Compression Conference 2001 |
| 2957 | Snowbird, Utah. 27-29 March 2001. |
| 2958 | </literallayout> |
| 2959 | |
| 2960 | </sect1> |
| 2961 | |
| 2962 | </chapter> |
| 2963 | |
| 2964 | </book> |