Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / zstd / 33d751999bbbf0ee4a50ade56012ea2759961e7e / . / doc / zstd_manual.html
blob: aac5af9c203a7c71484e453f57624a7261ad8a6d [file] [log] [blame]
Przemyslaw Skibinski86d94242016-10-24 16:07:53 +0200[diff] [blame]1<html>
2<head>
3<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
4<title>zstd 1.1.1 Manual</title>
5</head>
6<body>
7<h1>zstd 1.1.1 Manual</h1>
8<hr>
9<a name="Contents"></a><h2>Contents</h2>
10<ol>
11<li><a href="#Chapter1">Introduction</a></li>
12<li><a href="#Chapter2">Version</a></li>
13<li><a href="#Chapter3">Simple API</a></li>
14<li><a href="#Chapter4">Explicit memory management</a></li>
15<li><a href="#Chapter5">Simple dictionary API</a></li>
16<li><a href="#Chapter6">Fast dictionary API</a></li>
17<li><a href="#Chapter7">Streaming</a></li>
18<li><a href="#Chapter8">Streaming compression - HowTo</a></li>
19<li><a href="#Chapter9">Streaming decompression - HowTo</a></li>
20<li><a href="#Chapter10">START OF ADVANCED AND EXPERIMENTAL FUNCTIONS</a></li>
21<li><a href="#Chapter11">Advanced types</a></li>
22<li><a href="#Chapter12">Advanced compression functions</a></li>
23<li><a href="#Chapter13">Advanced decompression functions</a></li>
24<li><a href="#Chapter14">Advanced streaming functions</a></li>
25<li><a href="#Chapter15">Buffer-less and synchronous inner streaming functions</a></li>
26<li><a href="#Chapter16">Buffer-less streaming compression (synchronous mode)</a></li>
27<li><a href="#Chapter17">Buffer-less streaming decompression (synchronous mode)</a></li>
28<li><a href="#Chapter18">Block functions</a></li>
29</ol>
30<hr>
31<a name="Chapter1"></a><h2>Introduction</h2><pre>
32 Zstd, short for Zstandard, is a fast lossless compression algorithm, targeting real-time compression scenarios
33 at zlib-level and better compression ratios. The zstd compression library provides in-memory compression and
34 decompression functions. The library supports compression levels from 1 up to ZSTD_maxCLevel() which is 22.
35 Levels from 20 to 22 should be used with caution as they require about 300-1300 MB for compression.
36 Compression can be done in:
37 - a single step (described as Simple API)
38 - a single step, reusing a context (described as Explicit memory management)
39 - repeated calls of the compression function (described as Streaming compression)
40 The compression ratio achievable on small data can be highly improved using compression with a dictionary in:
41 - a single step (described as Simple dictionary API)
42 - a single step, reusing a dictionary (described as Fast dictionary API)
43
44 Advanced and experimantal functions can be accessed using #define ZSTD_STATIC_LINKING_ONLY before including zstd.h.
45 These APIs shall never be used with a dynamic library.
46 They are not "stable", their definition may change in the future. Only static linking is allowed.
47<BR></pre>
48
49<a name="Chapter2"></a><h2>Version</h2><pre></pre>
50
51<pre><b>unsigned ZSTD_versionNumber (void); </b>/**< returns version number of ZSTD */<b>
52</b></pre><BR>
53<a name="Chapter3"></a><h2>Simple API</h2><pre></pre>
54
55<pre><b>size_t ZSTD_compress( void* dst, size_t dstCapacity,
56 const void* src, size_t srcSize,
57 int compressionLevel);
58</b><p> Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
59 Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
60 @return : compressed size written into `dst` (<= `dstCapacity),
61 or an error code if it fails (which can be tested using ZSTD_isError())
62</p></pre><BR>
63
64<pre><b>size_t ZSTD_decompress( void* dst, size_t dstCapacity,
65 const void* src, size_t compressedSize);
66</b><p> `compressedSize` : must be the _exact_ size of a single compressed frame.
67 `dstCapacity` is an upper bound of originalSize.
68 If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data.
69 @return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
70 or an errorCode if it fails (which can be tested using ZSTD_isError())
71</p></pre><BR>
72
73<pre><b>unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
74</b><p> 'src' is the start of a zstd compressed frame.
75 @return : content size to be decompressed, as a 64-bits value _if known_, 0 otherwise.
76 note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode.
77 When `return==0`, data to decompress could be any size.
78 In which case, it's necessary to use streaming mode to decompress data.
79 Optionally, application can still use ZSTD_decompress() while relying on implied limits.
80 (For example, data may be necessarily cut into blocks <= 16 KB).
81 note 2 : decompressed size is always present when compression is done with ZSTD_compress()
82 note 3 : decompressed size can be very large (64-bits value),
83 potentially larger than what local system can handle as a single memory segment.
84 In which case, it's necessary to use streaming mode to decompress data.
85 note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified.
86 Always ensure result fits within application's authorized limits.
87 Each application can set its own limits.
88 note 5 : when `return==0`, if precise failure cause is needed, use ZSTD_getFrameParams() to know more.
89</p></pre><BR>
90
91<h3>Helper functions</h3><pre><b>int ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b>
92size_t ZSTD_compressBound(size_t srcSize); </b>/*!< maximum compressed size in worst case scenario */<b>
93unsigned ZSTD_isError(size_t code); </b>/*!< tells if a `size_t` function result is an error code */<b>
94const char* ZSTD_getErrorName(size_t code); </b>/*!< provides readable string from an error code */<b>
95</b></pre><BR>
96<a name="Chapter4"></a><h2>Explicit memory management</h2><pre></pre>
97
98<h3>Compression context</h3><pre><b>typedef struct ZSTD_CCtx_s ZSTD_CCtx;
99ZSTD_CCtx* ZSTD_createCCtx(void);
100size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx);
101</b></pre><BR>
102<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel);
103</b><p> Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see ZSTD_createCCtx())
104</p></pre><BR>
105
106<h3>Decompression context</h3><pre><b>typedef struct ZSTD_DCtx_s ZSTD_DCtx;
107ZSTD_DCtx* ZSTD_createDCtx(void);
108size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx);
109</b></pre><BR>
110<pre><b>size_t ZSTD_decompressDCtx(ZSTD_DCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
111</b><p> Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see ZSTD_createDCtx())
112</p></pre><BR>
113
114<a name="Chapter5"></a><h2>Simple dictionary API</h2><pre></pre>
115
116<pre><b>size_t ZSTD_compress_usingDict(ZSTD_CCtx* ctx,
117 void* dst, size_t dstCapacity,
118 const void* src, size_t srcSize,
119 const void* dict,size_t dictSize,
120 int compressionLevel);
121</b><p> Compression using a predefined Dictionary (see dictBuilder/zdict.h).
122 Note : This function load the dictionary, resulting in significant startup delay.
123</p></pre><BR>
124
125<pre><b>size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
126 void* dst, size_t dstCapacity,
127 const void* src, size_t srcSize,
128 const void* dict,size_t dictSize);
129</b><p> Decompression using a predefined Dictionary (see dictBuilder/zdict.h).
130 Dictionary must be identical to the one used during compression.
131 Note : This function load the dictionary, resulting in significant startup delay
132</p></pre><BR>
133
134<a name="Chapter6"></a><h2>Fast dictionary API</h2><pre></pre>
135
136<pre><b>ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel);
137</b><p> Create a digested dictionary, ready to start compression operation without startup delay.
138 `dict` can be released after ZSTD_CDict creation
139</p></pre><BR>
140
141<pre><b>size_t ZSTD_freeCDict(ZSTD_CDict* CDict);
142</b><p> Function frees memory allocated with ZSTD_createCDict()
143</p></pre><BR>
144
145<pre><b>size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
146 void* dst, size_t dstCapacity,
147 const void* src, size_t srcSize,
148 const ZSTD_CDict* cdict);
149</b><p> Compression using a digested Dictionary.
150 Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times.
151 Note that compression level is decided during dictionary creation
152</p></pre><BR>
153
154<pre><b>ZSTD_DDict* ZSTD_createDDict(const void* dict, size_t dictSize);
155</b><p> Create a digested dictionary, ready to start decompression operation without startup delay.
156 `dict` can be released after creation
157</p></pre><BR>
158
159<pre><b>size_t ZSTD_freeDDict(ZSTD_DDict* ddict);
160</b><p> Function frees memory allocated with ZSTD_createDDict()
161</p></pre><BR>
162
163<pre><b>size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
164 void* dst, size_t dstCapacity,
165 const void* src, size_t srcSize,
166 const ZSTD_DDict* ddict);
167</b><p> Decompression using a digested Dictionary
168 Faster startup than ZSTD_decompress_usingDict(), recommended when same dictionary is used multiple times.
169</p></pre><BR>
170
171<a name="Chapter7"></a><h2>Streaming</h2><pre></pre>
172
173<pre><b>typedef struct ZSTD_inBuffer_s {
174 const void* src; </b>/**< start of input buffer */<b>
175 size_t size; </b>/**< size of input buffer */<b>
176 size_t pos; </b>/**< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size */<b>
177} ZSTD_inBuffer;
178</b></pre><BR>
179<pre><b>typedef struct ZSTD_outBuffer_s {
180 void* dst; </b>/**< start of output buffer */<b>
181 size_t size; </b>/**< size of output buffer */<b>
182 size_t pos; </b>/**< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size */<b>
183} ZSTD_outBuffer;
184</b></pre><BR>
185<a name="Chapter8"></a><h2>Streaming compression - HowTo</h2><pre>
186 A ZSTD_CStream object is required to track streaming operation.
187 Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
188 ZSTD_CStream objects can be reused multiple times on consecutive compression operations.
189
190 Start by initializing ZSTD_CStream.
191 Use ZSTD_initCStream() to start a new compression operation.
192 Use ZSTD_initCStream_usingDict() for a compression which requires a dictionary.
193
194 Use ZSTD_compressStream() repetitively to consume input stream.
195 The function will automatically update both `pos` fields.
196 Note that it may not consume the entire input, in which case `pos < size`,
197 and it's up to the caller to present again remaining data.
198 @return : a size hint, preferred nb of bytes to use as input for next function call
199 (it's just a hint, to help latency a little, any other value will work fine)
200 (note : the size hint is guaranteed to be <= ZSTD_CStreamInSize() )
201 or an error code, which can be tested using ZSTD_isError().
202
203 At any moment, it's possible to flush whatever data remains within buffer, using ZSTD_flushStream().
204 `output->pos` will be updated.
205 Note some content might still be left within internal buffer if `output->size` is too small.
206 @return : nb of bytes still present within internal buffer (0 if it's empty)
207 or an error code, which can be tested using ZSTD_isError().
208
209 ZSTD_endStream() instructs to finish a frame.
210 It will perform a flush and write frame epilogue.
211 The epilogue is required for decoders to consider a frame completed.
212 Similar to ZSTD_flushStream(), it may not be able to flush the full content if `output->size` is too small.
213 In which case, call again ZSTD_endStream() to complete the flush.
214 @return : nb of bytes still present within internal buffer (0 if it's empty)
215 or an error code, which can be tested using ZSTD_isError().
216
217
218<BR></pre>
219
220<h3>Streaming compression functions</h3><pre><b>typedef struct ZSTD_CStream_s ZSTD_CStream;
221ZSTD_CStream* ZSTD_createCStream(void);
222size_t ZSTD_freeCStream(ZSTD_CStream* zcs);
223size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel);
224size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
225size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
226size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
227</b></pre><BR>
228<pre><b>size_t ZSTD_CStreamInSize(void); </b>/**< recommended size for input buffer */<b>
229</b></pre><BR>
230<pre><b>size_t ZSTD_CStreamOutSize(void); </b>/**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block in all circumstances. */<b>
231</b></pre><BR>
232<a name="Chapter9"></a><h2>Streaming decompression - HowTo</h2><pre>
233 A ZSTD_DStream object is required to track streaming operations.
234 Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
235 ZSTD_DStream objects can be re-used multiple times.
236
237 Use ZSTD_initDStream() to start a new decompression operation,
238 or ZSTD_initDStream_usingDict() if decompression requires a dictionary.
239 @return : recommended first input size
240
241 Use ZSTD_decompressStream() repetitively to consume your input.
242 The function will update both `pos` fields.
243 If `input.pos < input.size`, some input has not been consumed.
244 It's up to the caller to present again remaining data.
245 If `output.pos < output.size`, decoder has flushed everything it could.
246 @return : 0 when a frame is completely decoded and fully flushed,
247 an error code, which can be tested using ZSTD_isError(),
248 any other value > 0, which means there is still some work to do to complete the frame.
249 The return value is a suggested next input size (just an hint, to help latency).
250
251<BR></pre>
252
253<h3>Streaming decompression functions</h3><pre><b>typedef struct ZSTD_DStream_s ZSTD_DStream;
254ZSTD_DStream* ZSTD_createDStream(void);
255size_t ZSTD_freeDStream(ZSTD_DStream* zds);
256size_t ZSTD_initDStream(ZSTD_DStream* zds);
257size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
258</b></pre><BR>
259<pre><b>size_t ZSTD_DStreamInSize(void); </b>/*!< recommended size for input buffer */<b>
260</b></pre><BR>
261<pre><b>size_t ZSTD_DStreamOutSize(void); </b>/*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */<b>
262</b></pre><BR>
263<a name="Chapter10"></a><h2>START OF ADVANCED AND EXPERIMENTAL FUNCTIONS</h2><pre> The definitions in this section are considered experimental.
264 They should never be used with a dynamic library, as they may change in the future.
265 They are provided for advanced usages.
266 Use them only in association with static linking.
267
268<BR></pre>
269
270<a name="Chapter11"></a><h2>Advanced types</h2><pre></pre>
271
272<pre><b>typedef enum { ZSTD_fast, ZSTD_dfast, ZSTD_greedy, ZSTD_lazy, ZSTD_lazy2, ZSTD_btlazy2, ZSTD_btopt } ZSTD_strategy; </b>/* from faster to stronger */<b>
273</b></pre><BR>
274<pre><b>typedef struct {
275 unsigned windowLog; </b>/**< largest match distance : larger == more compression, more memory needed during decompression */<b>
276 unsigned chainLog; </b>/**< fully searched segment : larger == more compression, slower, more memory (useless for fast) */<b>
277 unsigned hashLog; </b>/**< dispatch table : larger == faster, more memory */<b>
278 unsigned searchLog; </b>/**< nb of searches : larger == more compression, slower */<b>
279 unsigned searchLength; </b>/**< match length searched : larger == faster decompression, sometimes less compression */<b>
280 unsigned targetLength; </b>/**< acceptable match size for optimal parser (only) : larger == more compression, slower */<b>
281 ZSTD_strategy strategy;
282} ZSTD_compressionParameters;
283</b></pre><BR>
284<pre><b>typedef struct {
285 unsigned contentSizeFlag; </b>/**< 1: content size will be in frame header (if known). */<b>
286 unsigned checksumFlag; </b>/**< 1: will generate a 22-bits checksum at end of frame, to be used for error detection by decompressor */<b>
287 unsigned noDictIDFlag; </b>/**< 1: no dict ID will be saved into frame header (if dictionary compression) */<b>
288} ZSTD_frameParameters;
289</b></pre><BR>
290<pre><b>typedef struct {
291 ZSTD_compressionParameters cParams;
292 ZSTD_frameParameters fParams;
293} ZSTD_parameters;
294</b></pre><BR>
295<h3>Custom memory allocation functions</h3><pre><b>typedef void* (*ZSTD_allocFunction) (void* opaque, size_t size);
296typedef void (*ZSTD_freeFunction) (void* opaque, void* address);
297typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem;
298</b></pre><BR>
299<a name="Chapter12"></a><h2>Advanced compression functions</h2><pre></pre>
300
301<pre><b>size_t ZSTD_estimateCCtxSize(ZSTD_compressionParameters cParams);
302</b><p> Gives the amount of memory allocated for a ZSTD_CCtx given a set of compression parameters.
303 `frameContentSize` is an optional parameter, provide `0` if unknown
304</p></pre><BR>
305
306<pre><b>ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem);
307</b><p> Create a ZSTD compression context using external alloc and free functions
308</p></pre><BR>
309
310<pre><b>size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
311</b><p> Gives the amount of memory used by a given ZSTD_CCtx
312</p></pre><BR>
313
314<pre><b>ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize,
315 ZSTD_parameters params, ZSTD_customMem customMem);
316</b><p> Create a ZSTD_CDict using external alloc and free, and customized compression parameters
317</p></pre><BR>
318
319<pre><b>size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
320</b><p> Gives the amount of memory used by a given ZSTD_sizeof_CDict
321</p></pre><BR>
322
323<pre><b>ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSize, size_t dictSize);
324</b><p> same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of a `ZSTD_compressionParameters`.
325 All fields of `ZSTD_frameParameters` are set to default (0)
326</p></pre><BR>
327
328<pre><b>ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSize, size_t dictSize);
329</b><p> @return ZSTD_compressionParameters structure for a selected compression level and srcSize.
330 `srcSize` value is optional, select 0 if not known
331</p></pre><BR>
332
333<pre><b>size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
334</b><p> Ensure param values remain within authorized range
335</p></pre><BR>
336
337<pre><b>ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);
338</b><p> optimize params for a given `srcSize` and `dictSize`.
339 both values are optional, select `0` if unknown.
340</p></pre><BR>
341
342<pre><b>size_t ZSTD_compress_advanced (ZSTD_CCtx* ctx,
343 void* dst, size_t dstCapacity,
344 const void* src, size_t srcSize,
345 const void* dict,size_t dictSize,
346 ZSTD_parameters params);
347</b><p> Same as ZSTD_compress_usingDict(), with fine-tune control of each compression parameter
348</p></pre><BR>
349
350<a name="Chapter13"></a><h2>Advanced decompression functions</h2><pre></pre>
351
352<pre><b>size_t ZSTD_estimateDCtxSize(void);
353</b><p> Gives the potential amount of memory allocated to create a ZSTD_DCtx
354</p></pre><BR>
355
356<pre><b>ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem);
357</b><p> Create a ZSTD decompression context using external alloc and free functions
358</p></pre><BR>
359
360<pre><b>size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
361</b><p> Gives the amount of memory used by a given ZSTD_DCtx
362</p></pre><BR>
363
364<pre><b>size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
365</b><p> Gives the amount of memory used by a given ZSTD_DDict
366</p></pre><BR>
367
368<a name="Chapter14"></a><h2>Advanced streaming functions</h2><pre></pre>
369
370<h3>Advanced Streaming compression functions</h3><pre><b>ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
371size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel);
372size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs, const void* dict, size_t dictSize,
373 ZSTD_parameters params, unsigned long long pledgedSrcSize); </b>/**< pledgedSrcSize is optional and can be zero == unknown */<b>
374size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize); </b>/**< re-use compression parameters from previous init; saves dictionary loading */<b>
375size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs);
376</b></pre><BR>
377<h3>Advanced Streaming decompression functions</h3><pre><b>typedef enum { ZSTDdsp_maxWindowSize } ZSTD_DStreamParameter_e;
378ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
379size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);
380size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue);
381size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression parameters from previous init; saves dictionary loading */<b>
382size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
383</b></pre><BR>
384<a name="Chapter15"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre>
385 This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
386 But it's also a complex one, with many restrictions (documented below).
387 Prefer using normal streaming API for an easier experience
388
389<BR></pre>
390
391<a name="Chapter16"></a><h2>Buffer-less streaming compression (synchronous mode)</h2><pre>
392 A ZSTD_CCtx object is required to track streaming operations.
393 Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource.
394 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
395
396 Start by initializing a context.
397 Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression,
398 or ZSTD_compressBegin_advanced(), for finer parameter control.
399 It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()
400
401 Then, consume your input using ZSTD_compressContinue().
402 There are some important considerations to keep in mind when using this advanced function :
403 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffer only.
404 - Interface is synchronous : input is consumed entirely and produce 1+ (or more) compressed blocks.
405 - Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
406 Worst case evaluation is provided by ZSTD_compressBound().
407 ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
408 - ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to maximum distance size, see WindowLog).
409 It remembers all previous contiguous blocks, plus one separated memory segment (which can itself consists of multiple contiguous blocks)
410 - ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
411 In which case, it will "discard" the relevant memory section from its history.
412
413 Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
414 It's possible to use a NULL,0 src content, in which case, it will write a final empty block to end the frame,
415 Without last block mark, frames will be considered unfinished (broken) by decoders.
416
417 You can then reuse `ZSTD_CCtx` (ZSTD_compressBegin()) to compress some new frame.
418<BR></pre>
419
420<h3>Buffer-less streaming compression functions</h3><pre><b>size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
421size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
422size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize);
423size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize);
424size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
425size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
426</b></pre><BR>
427<a name="Chapter17"></a><h2>Buffer-less streaming decompression (synchronous mode)</h2><pre>
428 A ZSTD_DCtx object is required to track streaming operations.
429 Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
430 A ZSTD_DCtx object can be re-used multiple times.
431
432 First typical operation is to retrieve frame parameters, using ZSTD_getFrameParams().
433 It fills a ZSTD_frameParams structure which provide important information to correctly decode the frame,
434 such as the minimum rolling buffer size to allocate to decompress data (`windowSize`),
435 and the dictionary ID used.
436 (Note : content size is optional, it may not be present. 0 means : content size unknown).
437 Note that these values could be wrong, either because of data malformation, or because an attacker is spoofing deliberate false information.
438 As a consequence, check that values remain within valid application range, especially `windowSize`, before allocation.
439 Each application can set its own limit, depending on local restrictions. For extended interoperability, it is recommended to support at least 8 MB.
440 Frame parameters are extracted from the beginning of the compressed frame.
441 Data fragment must be large enough to ensure successful decoding, typically `ZSTD_frameHeaderSize_max` bytes.
442 @result : 0 : successful decoding, the `ZSTD_frameParams` structure is correctly filled.
443 >0 : `srcSize` is too small, please provide at least @result bytes on next attempt.
444 errorCode, which can be tested using ZSTD_isError().
445
446 Start decompression, with ZSTD_decompressBegin() or ZSTD_decompressBegin_usingDict().
447 Alternatively, you can copy a prepared context, using ZSTD_copyDCtx().
448
449 Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively.
450 ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue().
451 ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail.
452
453 @result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
454 It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some metadata item.
455 It can also be an error code, which can be tested with ZSTD_isError().
456
457 ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize`.
458 They should preferably be located contiguously, prior to current block.
459 Alternatively, a round buffer of sufficient size is also possible. Sufficient size is determined by frame parameters.
460 ZSTD_decompressContinue() is very sensitive to contiguity,
461 if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place,
462 or that previous contiguous segment is large enough to properly handle maximum back-reference.
463
464 A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
465 Context can then be reset to start a new decompression.
466
467 Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInputType().
468 This information is not required to properly decode a frame.
469
470 == Special case : skippable frames ==
471
472 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
473 Skippable frames will be ignored (skipped) by a decompressor. The format of skippable frames is as follows :
474 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
475 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
476 c) Frame Content - any content (User Data) of length equal to Frame Size
477 For skippable frames ZSTD_decompressContinue() always returns 0.
478 For skippable frames ZSTD_getFrameParams() returns fparamsPtr->windowLog==0 what means that a frame is skippable.
479 It also returns Frame Size as fparamsPtr->frameContentSize.
480<BR></pre>
481
482<pre><b>typedef struct {
483 unsigned long long frameContentSize;
484 unsigned windowSize;
485 unsigned dictID;
486 unsigned checksumFlag;
487} ZSTD_frameParams;
488</b></pre><BR>
489<h3>Buffer-less streaming decompression functions</h3><pre><b>size_t ZSTD_getFrameParams(ZSTD_frameParams* fparamsPtr, const void* src, size_t srcSize); </b>/**< doesn't consume input, see details below */<b>
490size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx);
491size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
492void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
493size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx);
494size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
495typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
496ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
497</b></pre><BR>
498<a name="Chapter18"></a><h2>Block functions</h2><pre>
499 Block functions produce and decode raw zstd blocks, without frame metadata.
500 Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes).
501 User will have to take in charge required information to regenerate data, such as compressed and content sizes.
502
503 A few rules to respect :
504 - Compressing and decompressing require a context structure
505 + Use ZSTD_createCCtx() and ZSTD_createDCtx()
506 - It is necessary to init context before starting
507 + compression : ZSTD_compressBegin()
508 + decompression : ZSTD_decompressBegin()
509 + variants _usingDict() are also allowed
510 + copyCCtx() and copyDCtx() work too
511 - Block size is limited, it must be <= ZSTD_getBlockSizeMax()
512 + If you need to compress more, cut data into multiple blocks
513 + Consider using the regular ZSTD_compress() instead, as frame metadata costs become negligible when source size is large.
514 - When a block is considered not compressible enough, ZSTD_compressBlock() result will be zero.
515 In which case, nothing is produced into `dst`.
516 + User must test for such outcome and deal directly with uncompressed data
517 + ZSTD_decompressBlock() doesn't accept uncompressed data as input !!!
518 + In case of multiple successive blocks, decoder must be informed of uncompressed block existence to follow proper history.
519 Use ZSTD_insertBlock() in such a case.
520<BR></pre>
521
522<h3>Raw zstd block functions</h3><pre><b>size_t ZSTD_getBlockSizeMax(ZSTD_CCtx* cctx);
523size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
524size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
525size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); </b>/**< insert block into `dctx` history. Useful for uncompressed blocks */<b>
526</b></pre><BR>
527</html>
528</body>