zstandard 0.5.1

Simple API

compress(data) compresses and returns data as a one-shot operation.:

cctx = zstd.ZsdCompressor()
compressed = cctx.compress(b'data to compress')

Streaming Input API

write_to(fh) (which behaves as a context manager) allows you to stream data into a compressor.:

cctx = zstd.ZstdCompressor(level=10)
with cctx.write_to(fh) as compressor:
    compressor.write(b'chunk 0')
    compressor.write(b'chunk 1')
    ...

The argument to write_to() must have a write(data) method. As compressed data is available, write() will be called with the comrpessed data as its argument. Many common Python types implement write(), including open file handles and io.BytesIO.

write_to() returns an object representing a streaming compressor instance. It must be used as a context manager. That object’s write(data) method is used to feed data into the compressor.

If the size of the data being fed to this streaming compressor is known, you can declare it before compression begins:

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh, size=data_len) as compressor:
    compressor.write(chunk0)
    compressor.write(chunk1)
    ...

Declaring the size of the source data allows compression parameters to be tuned. And if write_content_size is used, it also results in the content size being written into the frame header of the output data.

The size of chunks being write() to the destination can be specified:

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh, write_size=32768) as compressor:
    ...

To see how much memory is being used by the streaming compressor:

cctx = zstd.ZstdCompressor()
with cctx.write_to(fh) as compressor:
    ...
    byte_size = compressor.memory_size()

Streaming Output API

read_from(reader) provides a mechanism to stream data out of a compressor as an iterator of data chunks.:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh):
     # Do something with emitted data.

read_from() accepts an object that has a read(size) method or conforms to the buffer protocol. (bytes and memoryview are 2 common types that provide the buffer protocol.)

Uncompressed data is fetched from the source either by calling read(size) or by fetching a slice of data from the object directly (in the case where the buffer protocol is being used). The returned iterator consists of chunks of compressed data.

Like write_to(), read_from() also accepts a size argument declaring the size of the input stream:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh, size=some_int):
    pass

You can also control the size that data is read() from the source and the ideal size of output chunks:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_from(fh, read_size=16384, write_size=8192):
    pass

Stream Copying API

copy_stream(ifh, ofh) can be used to copy data between 2 streams while compressing it.:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh)

For example, say you wish to compress a file:

cctx = zstd.ZstdCompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    cctx.copy_stream(ifh, ofh)

It is also possible to declare the size of the source stream:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, size=len_of_input)

You can also specify how large the chunks that are read() and write() from and to the streams:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, read_size=32768, write_size=16384)

The stream copier returns a 2-tuple of bytes read and written:

cctx = zstd.ZstdCompressor()
read_count, write_count = cctx.copy_stream(ifh, ofh)

Compressor API

compressobj() returns an object that exposes compress(data) and flush() methods. Each returns compressed data or an empty bytes.

The purpose of compressobj() is to provide an API-compatible interface with zlib.compressobj and bz2.BZ2Compressor. This allows callers to swap in different compressor objects while using the same API.

Once flush() is called, the compressor will no longer accept new data to compress(). flush() must be called to end the compression context. If not called, the returned data may be incomplete.

Here is how this API should be used:

cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj()
data = cobj.compress(b'raw input 0')
data = cobj.compress(b'raw input 1')
data = cobj.flush()

For best performance results, keep input chunks under 256KB. This avoids extra allocations for a large output object.

It is possible to declare the input size of the data that will be fed into the compressor:

cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj(size=6)
data = cobj.compress(b'foobar')
data = cobj.flush()

Simple API

decompress(data) can be used to decompress an entire compressed zstd frame in a single operation.:

dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress(data)

By default, decompress(data) will only work on data written with the content size encoded in its header. This can be achieved by creating a ZstdCompressor with write_content_size=True. If compressed data without an embedded content size is seen, zstd.ZstdError will be raised.

If the compressed data doesn’t have its content size embedded within it, decompression can be attempted by specifying the max_output_size argument.:

dctx = zstd.ZstdDecompressor()
uncompressed = dctx.decompress(data, max_output_size=1048576)

Ideally, max_output_size will be identical to the decompressed output size.

If max_output_size is too small to hold the decompressed data, zstd.ZstdError will be raised.

If max_output_size is larger than the decompressed data, the allocated output buffer will be resized to only use the space required.

Please note that an allocation of the requested max_output_size will be performed every time the method is called. Setting to a very large value could result in a lot of work for the memory allocator and may result in MemoryError being raised if the allocation fails.

If the exact size of decompressed data is unknown, it is strongly recommended to use a streaming API.

Streaming Input API

write_to(fh) can be used to incrementally send compressed data to a decompressor.:

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh) as decompressor:
    decompressor.write(compressed_data)

This behaves similarly to zstd.ZstdCompressor: compressed data is written to the decompressor by calling write(data) and decompressed output is written to the output object by calling its write(data) method.

The size of chunks being write() to the destination can be specified:

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh, write_size=16384) as decompressor:
    pass

You can see how much memory is being used by the decompressor:

dctx = zstd.ZstdDecompressor()
with dctx.write_to(fh) as decompressor:
    byte_size = decompressor.memory_size()

Streaming Output API

read_from(fh) provides a mechanism to stream decompressed data out of a compressed source as an iterator of data chunks.:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_from(fh):
    # Do something with original data.

read_from() accepts a) an object with a read(size) method that will return compressed bytes b) an object conforming to the buffer protocol that can expose its data as a contiguous range of bytes. The bytes and memoryview types expose this buffer protocol.

read_from() returns an iterator whose elements are chunks of the decompressed data.

The size of requested read() from the source can be specified:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_from(fh, read_size=16384):
    pass

It is also possible to skip leading bytes in the input data:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_from(fh, skip_bytes=1):
    pass

Skipping leading bytes is useful if the source data contains extra header data but you want to avoid the overhead of making a buffer copy or allocating a new memoryview object in order to decompress the data.

Similarly to ZstdCompressor.read_from(), the consumer of the iterator controls when data is decompressed. If the iterator isn’t consumed, decompression is put on hold.

When read_from() is passed an object conforming to the buffer protocol, the behavior may seem similar to what occurs when the simple decompression API is used. However, this API works when the decompressed size is unknown. Furthermore, if feeding large inputs, the decompressor will work in chunks instead of performing a single operation.

Stream Copying API

copy_stream(ifh, ofh) can be used to copy data across 2 streams while performing decompression.:

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh)

e.g. to decompress a file to another file:

dctx = zstd.ZstdDecompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    dctx.copy_stream(ifh, ofh)

The size of chunks being read() and write() from and to the streams can be specified:

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh, read_size=8192, write_size=16384)

Decompressor API

decompressobj() returns an object that exposes a decompress(data) methods. Compressed data chunks are fed into decompress(data) and uncompressed output (or an empty bytes) is returned. Output from subsequent calls needs to be concatenated to reassemble the full decompressed byte sequence.

The purpose of decompressobj() is to provide an API-compatible interface with zlib.decompressobj and bz2.BZ2Decompressor. This allows callers to swap in different decompressor objects while using the same API.

Each object is single use: once an input frame is decoded, decompress() can no longer be called.

Here is how this API should be used:

dctx = zstd.ZstdDeompressor()
dobj = cctx.decompressobj()
data = dobj.decompress(compressed_chunk_0)
data = dobj.decompress(compressed_chunk_1)

estimate_compression_context_size(CompressionParameters)

Given a CompressionParameters struct, estimate the memory size required to perform compression.

estimate_decompression_context_size()

Estimate the memory size requirements for a decompressor instance.