New feature: "Large Window Brotli" (#640)
* New feature: "Large Window Brotli"
By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.
Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).
Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.
Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.
Other C changes:
- combined / moved `context.h` to `common`
- moved transforms to `common`
- unified some aspects of code formatting
- added an abstraction for encoder (static) dictionary
- moved default allocator/deallocator functions to `common`
brotli CLI:
- window size is auto-adjusted if not specified explicitly
Java:
- added "eager" decoding both to JNI wrapper and pure decoder
- huge speed-up of `DictionaryData` initialization
* Add dictionaryless compressed dictionary
* Fix `sources.lst`
* Fix `sources.lst` and add a note that `libtool` is also required.
* Update setup.py
* Fix `EagerStreamTest`
* Fix BUILD file
* Add missing `libdivsufsort` dependency
* Fix "unused parameter" warning.
diff --git a/setup.py b/setup.py
index a8a2ebe..d8478b3 100644
--- a/setup.py
+++ b/setup.py
@@ -182,6 +182,7 @@
sources=[
'python/_brotli.cc',
'c/common/dictionary.c',
+ 'c/common/transform.c',
'c/dec/bit_reader.c',
'c/dec/decode.c',
'c/dec/huffman.c',
@@ -196,6 +197,7 @@
'c/enc/compress_fragment_two_pass.c',
'c/enc/dictionary_hash.c',
'c/enc/encode.c',
+ 'c/enc/encoder_dict.c',
'c/enc/entropy_encode.c',
'c/enc/histogram.c',
'c/enc/literal_cost.c',
@@ -206,15 +208,15 @@
],
depends=[
'c/common/constants.h',
+ 'c/common/context.h',
'c/common/dictionary.h',
+ 'c/common/platform.h',
+ 'c/common/transform.h',
'c/common/version.h',
'c/dec/bit_reader.h',
- 'c/dec/context.h',
'c/dec/huffman.h',
- 'c/dec/port.h',
'c/dec/prefix.h',
'c/dec/state.h',
- 'c/dec/transform.h',
'c/enc/backward_references.h',
'c/enc/backward_references_hq.h',
'c/enc/backward_references_inc.h',
@@ -229,8 +231,8 @@
'c/enc/command.h',
'c/enc/compress_fragment.h',
'c/enc/compress_fragment_two_pass.h',
- 'c/enc/context.h',
'c/enc/dictionary_hash.h',
+ 'c/enc/encoder_dict.h',
'c/enc/entropy_encode.h',
'c/enc/entropy_encode_static.h',
'c/enc/fast_log.h',
@@ -247,7 +249,6 @@
'c/enc/memory.h',
'c/enc/metablock.h',
'c/enc/metablock_inc.h',
- 'c/enc/port.h',
'c/enc/prefix.h',
'c/enc/quality.h',
'c/enc/ringbuffer.h',