zstandard 0.3.2

Expected Changes

The author is reasonably confident in the current state of what’s implemented on the ZstdCompressor and ZstdDecompressor types. Those APIs likely won’t change significantly. Some low-level behavior (such as naming and types expected by arguments) may change.

There will likely be arguments added to control the input and output buffer sizes (currently, certain operations read and write in chunk sizes using zstd’s preferred defaults).

There should be an API that accepts an object that conforms to the buffer interface and returns an iterator over compressed or decompressed output.

The author is on the fence as to whether to support the extremely low level compression and decompression APIs. It could be useful to support compression without the framing headers. But the author doesn’t believe it a high priority at this time.

The CFFI bindings are half-baked and need to be finished.

ZstdCompressor

The ZstdCompressor class provides an interface for performing compression operations.

Each instance is associated with parameters that control compression behavior. These come from the following named arguments (all optional):

level

Integer compression level. Valid values are between 1 and 22.

dict_data

Compression dictionary to use.

Note: When using dictionary data and compress() is called multiple times, the CompressionParameters derived from an integer compression level and the first compressed data’s size will be reused for all subsequent operations. This may not be desirable if source data size varies significantly.

compression_params

A CompressionParameters instance (overrides the level value).

write_checksum

Whether a 4 byte checksum should be written with the compressed data. Defaults to False. If True, the decompressor can verify that decompressed data matches the original input data.

write_content_size

Whether the size of the uncompressed data will be written into the header of compressed data. Defaults to False. The data will only be written if the compressor knows the size of the input data. This is likely not true for streaming compression.

write_dict_id

Whether to write the dictionary ID into the compressed data. Defaults to True. The dictionary ID is only written if a dictionary is being used.

cctx = zstd.ZsdCompressor()
compressed = cctx.compress(b'data to compress')

cctx = zstd.ZstdCompressor(level=10)
with cctx.write_to(fh) as compressor:
    compressor.write(b'chunk 0')
        compressor.write(b'chunk 1')
        ...

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh, size=data_len) as compressor:
    compressor.write(chunk0)
        compressor.write(chunk1)
        ...

cctx = zstd.ZstdCompressor()
    with cctx.write_to(fh) as compressor:
        ...
            byte_size = compressor.memory_size()

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh):
     # Do something with emitted data.

cctx = zstd.ZstdCompressor()
    for chunk in cctx.read_from(fh, size=some_int):
        pass

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh)

cctx = zstd.ZstdCompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
        cctx.copy_stream(ifh, ofh)

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, size=len_of_input)

cctx = zstd.ZstdCompressor()
read_count, write_count = cctx.copy_stream(ifh, ofh)

ZstdDecompressor

The ZstdDecompressor class provides an interface for performing decompression.

Each instance is associated with parameters that control decompression. These come from the following named arguments (all optional):

dict_data

Compression dictionary to use.

The interface of this class is very similar to ZstdCompressor (by design).

dctx = zstd.ZstdDecompressor()
    decompressed = dctx.decompress(data)

dctx = zstd.ZstdDecompressor()
    uncompressed = dctx.decompress(data, max_output_size=1048576)

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh) as decompressor:
    decompressor.write(compressed_data)

dctx = zstd.ZstdDecompressor()
    with dctx.write_to(fh) as decompressor:
        byte_size = decompressor.memory_size()

dctx = zstd.ZstdDecompressor()
    for chunk in dctx.read_from(fh):
        # Do something with original data.

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh)

dctx = zstd.ZstdDecompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    dctx.copy_stream(ifh, ofh)

Choosing an API

Various forms of compression and decompression APIs are provided because each are suitable for different use cases.

The simple/one-shot APIs are useful for small data, when the decompressed data size is known (either recorded in the zstd frame header via write_content_size or known via an out-of-band mechanism, such as a file size).

A limitation of the simple APIs is that input or output data must fit in memory. And unless using advanced tricks with Python buffer objects, both input and output must fit in memory simultaneously.

Another limitation is that compression or decompression is performed as a single operation. So if you feed large input, it could take a long time for the function to return.

The streaming APIs do not have the limitations of the simple API. The cost to this is they are more complex to use than a single function call.

The streaming APIs put the caller in control of compression and decompression behavior by allowing them to directly control either the input or output side of the operation.

With the streaming input APIs, the caller feeds data into the compressor or decompressor as they see fit. Output data will only be written after the caller has explicitly written data.

With the streaming output APIs, the caller consumes output from the compressor or decompressor as they see fit. The compressor or decompressor will only consume data from the source when the caller is ready to receive it.

One end of the streaming APIs involves a file-like object that must write() output data or read() input data. Depending on what the backing storage for these objects is, those operations may not complete quickly. For example, when streaming compressed data to a file, the write() into a streaming compressor could result in a write() to the filesystem, which may take a long time to finish due to slow I/O on the filesystem. So, there may be overhead in streaming APIs beyond the compression and decompression operations.

Dictionary Creation and Management

Zstandard allows dictionaries to be used when compressing and decompressing data. The idea is that if you are compressing a lot of similar data, you can precompute common properties of that data (such as recurring byte sequences) to achieve better compression ratios.

In Python, compression dictionaries are represented as the ZstdCompressionDict type.

Instances can be constructed from bytes:

dict_data = zstd.ZstdCompressionDict(data)

More interestingly, instances can be created by training on sample data:

dict_data = zstd.train_dictionary(size, samples)

This takes a list of bytes instances and creates and returns a ZstdCompressionDict.

You can see how many bytes are in the dictionary by calling len():

dict_data = zstd.train_dictionary(size, samples)
dict_size = len(dict_data)  # will not be larger than ``size``

Once you have a dictionary, you can pass it to the objects performing compression and decompression:

dict_data = zstd.train_dictionary(16384, samples)

cctx = zstd.ZstdCompressor(dict_data=data)
for source_data in input_data:
    compressed = cctx.compress(source_data)
        # Do something with compressed data.

dctx = zstd.ZstdDecompressor(dict_data=dict_data)
for compressed_data in input_data:
    buffer = io.BytesIO()
    with dctx.write_to(buffer) as decompressor:
            decompressor.write(compressed_data)
        # Do something with raw data in ``buffer``.

Dictionaries have unique integer IDs. You can retrieve this ID via:

dict_id = zstd.dictionary_id(dict_data)

Explicit Compression Parameters

Zstandard’s integer compression levels along with the input size and dictionary size are converted into a data structure defining multiple parameters to tune behavior of the compression algorithm. It is possible to use define this data structure explicitly to have lower-level control over compression behavior.

The zstd.CompressionParameters type represents this data structure. You can see how Zstandard converts compression levels to this data structure by calling zstd.get_compression_parameters(). e.g.:

params = zstd.get_compression_parameters(5)

This function also accepts the uncompressed data size and dictionary size to adjust parameters:

params = zstd.get_compression_parameters(3, source_size=len(data), dict_size=len(dict_data))

You can also construct compression parameters from their low-level components:

params = zstd.CompressionParameters(20, 6, 12, 5, 4, 10, zstd.STRATEGY_FAST)

You can then configure a compressor to use the custom parameters:

cctx = zstd.ZstdCompressor(compression_params=params)

The members of the CompressionParameters tuple are as follows:

* 0 - Window log
* 1 - Chain log
* 2 - Hash log
* 3 - Search log
* 4 - Search length
* 5 - Target length
* 6 - Strategy (one of the ``zstd.STRATEGY_`` constants)

You’ll need to read the Zstandard documentation for what these parameters do.

Misc Functionality

Constants

The following module constants/attributes are exposed:

ZSTD_VERSION

This module attribute exposes a 3-tuple of the Zstandard version. e.g. (1, 0, 0)

MAX_COMPRESSION_LEVEL

Integer max compression level accepted by compression functions

COMPRESSION_RECOMMENDED_INPUT_SIZE

Recommended chunk size to feed to compressor functions

COMPRESSION_RECOMMENDED_OUTPUT_SIZE

Recommended chunk size for compression output

DECOMPRESSION_RECOMMENDED_INPUT_SIZE

Recommended chunk size to feed into decompresor functions

DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE

Recommended chunk size for decompression output

FRAME_HEADER

bytes containing header of the Zstandard frame

MAGIC_NUMBER

Frame header as an integer

WINDOWLOG_MIN

Minimum value for compression parameter

WINDOWLOG_MAX

Maximum value for compression parameter

CHAINLOG_MIN

Minimum value for compression parameter

CHAINLOG_MAX

Maximum value for compression parameter

HASHLOG_MIN

Minimum value for compression parameter

HASHLOG_MAX

Maximum value for compression parameter

SEARCHLOG_MIN

Minimum value for compression parameter

SEARCHLOG_MAX

Maximum value for compression parameter

SEARCHLENGTH_MIN

Minimum value for compression parameter

SEARCHLENGTH_MAX

Maximum value for compression parameter

TARGETLENGTH_MIN

Minimum value for compression parameter

TARGETLENGTH_MAX

Maximum value for compression parameter

STRATEGY_FAST

Compression strategory

STRATEGY_DFAST

Compression strategory

STRATEGY_GREEDY

Compression strategory

STRATEGY_LAZY

Compression strategory

STRATEGY_LAZY2

Compression strategory

STRATEGY_BTLAZY2

Compression strategory

STRATEGY_BTOPT

Compression strategory