zstandard 0.9.1

Utility Methods

frame_progression() returns a 3-tuple containing the number of bytes ingested, consumed, and produced by the current compression operation.

memory_size() obtains the memory utilization of the underlying zstd compression context, in bytes.:

cctx = zstd.ZstdCompressor()
memory = cctx.memory_size()

Simple API

compress(data) compresses and returns data as a one-shot operation.:

cctx = zstd.ZstdCompressor()
compressed = cctx.compress(b'data to compress')

The data argument can be any object that implements the buffer protocol.

Stream Reader API

stream_reader(source) can be used to obtain an object conforming to the io.RawIOBase interface for reading compressed output as a stream:

with open(path, 'rb') as fh:
    cctx = zstd.ZstdCompressor()
    with cctx.stream_reader(fh) as reader:
        while True:
            chunk = reader.read(16384)
            if not chunk:
                break

            # Do something with compressed chunk.

The stream can only be read within a context manager. When the context manager exits, the stream is closed and the underlying resource is released and future operations against the compression stream stream will fail.

The source argument to stream_reader() can be any object with a read(size) method or any object implementing the buffer protocol.

stream_reader() accepts a size argument specifying how large the input stream is. This is used to adjust compression parameters so they are tailored to the source size.:

with open(path, 'rb') as fh:
    cctx = zstd.ZstdCompressor()
    with cctx.stream_reader(fh, size=os.stat(path).st_size) as reader:
        ...

If the source is a stream, you can specify how large read() requests to that stream should be via the read_size argument. It defaults to zstandard.COMPRESSION_RECOMMENDED_INPUT_SIZE.:

with open(path, 'rb') as fh:
    cctx = zstd.ZstdCompressor()
    # Will perform fh.read(8192) when obtaining data to feed into the
    # compressor.
    with cctx.stream_reader(fh, read_size=8192) as reader:
        ...

The stream returned by stream_reader() is neither writable nor seekable (even if the underlying source is seekable). readline() and readlines() are not implemented because they don’t make sense for compressed data. tell() returns the number of compressed bytes emitted so far.

Streaming Input API

stream_writer(fh) (which behaves as a context manager) allows you to stream data into a compressor.:

cctx = zstd.ZstdCompressor(level=10)
with cctx.stream_writer(fh) as compressor:
    compressor.write(b'chunk 0')
    compressor.write(b'chunk 1')
    ...

The argument to stream_writer() must have a write(data) method. As compressed data is available, write() will be called with the compressed data as its argument. Many common Python types implement write(), including open file handles and io.BytesIO.

stream_writer() returns an object representing a streaming compressor instance. It must be used as a context manager. That object’s write(data) method is used to feed data into the compressor.

A flush() method can be called to evict whatever data remains within the compressor’s internal state into the output object. This may result in 0 or more write() calls to the output object.

Both write() and flush() return the number of bytes written to the object’s write(). In many cases, small inputs do not accumulate enough data to cause a write and write() will return 0.

If the size of the data being fed to this streaming compressor is known, you can declare it before compression begins:

cctx = zstd.ZstdCompressor()
with cctx.stream_writer(fh, size=data_len) as compressor:
    compressor.write(chunk0)
    compressor.write(chunk1)
    ...

Declaring the size of the source data allows compression parameters to be tuned. And if write_content_size is used, it also results in the content size being written into the frame header of the output data.

The size of chunks being write() to the destination can be specified:

cctx = zstd.ZstdCompressor()
with cctx.stream_writer(fh, write_size=32768) as compressor:
    ...

To see how much memory is being used by the streaming compressor:

cctx = zstd.ZstdCompressor()
with cctx.stream_writer(fh) as compressor:
    ...
    byte_size = compressor.memory_size()

Thte total number of bytes written so far are exposed via tell():

cctx = zstd.ZstdCompressor()
with cctx.stream_writer(fh) as compressor:
    ...
    total_written = compressor.tell()

Streaming Output API

read_to_iter(reader) provides a mechanism to stream data out of a compressor as an iterator of data chunks.:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_to_iter(fh):
     # Do something with emitted data.

read_to_iter() accepts an object that has a read(size) method or conforms to the buffer protocol.

Uncompressed data is fetched from the source either by calling read(size) or by fetching a slice of data from the object directly (in the case where the buffer protocol is being used). The returned iterator consists of chunks of compressed data.

If reading from the source via read(), read() will be called until it raises or returns an empty bytes (b''). It is perfectly valid for the source to deliver fewer bytes than were what requested by read(size).

Like stream_writer(), read_to_iter() also accepts a size argument declaring the size of the input stream:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_to_iter(fh, size=some_int):
    pass

You can also control the size that data is read() from the source and the ideal size of output chunks:

cctx = zstd.ZstdCompressor()
for chunk in cctx.read_to_iter(fh, read_size=16384, write_size=8192):
    pass

Unlike stream_writer(), read_to_iter() does not give direct control over the sizes of chunks fed into the compressor. Instead, chunk sizes will be whatever the object being read from delivers. These will often be of a uniform size.

Stream Copying API

copy_stream(ifh, ofh) can be used to copy data between 2 streams while compressing it.:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh)

For example, say you wish to compress a file:

cctx = zstd.ZstdCompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    cctx.copy_stream(ifh, ofh)

It is also possible to declare the size of the source stream:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, size=len_of_input)

You can also specify how large the chunks that are read() and write() from and to the streams:

cctx = zstd.ZstdCompressor()
cctx.copy_stream(ifh, ofh, read_size=32768, write_size=16384)

The stream copier returns a 2-tuple of bytes read and written:

cctx = zstd.ZstdCompressor()
read_count, write_count = cctx.copy_stream(ifh, ofh)

Compressor API

compressobj() returns an object that exposes compress(data) and flush() methods. Each returns compressed data or an empty bytes.

The purpose of compressobj() is to provide an API-compatible interface with zlib.compressobj, bz2.BZ2Compressor, etc. This allows callers to swap in different compressor objects while using the same API.

flush() accepts an optional argument indicating how to end the stream. zstd.COMPRESSOBJ_FLUSH_FINISH (the default) ends the compression stream. Once this type of flush is performed, compress() and flush() can no longer be called. This type of flush must be called to end the compression context. If not called, returned data may be incomplete.

A zstd.COMPRESSOBJ_FLUSH_BLOCK argument to flush() will flush a zstd block. Flushes of this type can be performed multiple times. The next call to compress() will begin a new zstd block.

Here is how this API should be used:

cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj()
data = cobj.compress(b'raw input 0')
data = cobj.compress(b'raw input 1')
data = cobj.flush()

Or to flush blocks:

cctx.zstd.ZstdCompressor()
cobj = cctx.compressobj()
data = cobj.compress(b'chunk in first block')
data = cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK)
data = cobj.compress(b'chunk in second block')
data = cobj.flush()

For best performance results, keep input chunks under 256KB. This avoids extra allocations for a large output object.

It is possible to declare the input size of the data that will be fed into the compressor:

cctx = zstd.ZstdCompressor()
cobj = cctx.compressobj(size=6)
data = cobj.compress(b'foobar')
data = cobj.flush()

Batch Compression API

(Experimental. Not yet supported in CFFI bindings.)

multi_compress_to_buffer(data, [threads=0]) performs compression of multiple inputs as a single operation.

Data to be compressed can be passed as a BufferWithSegmentsCollection, a BufferWithSegments, or a list containing byte like objects. Each element of the container will be compressed individually using the configured parameters on the ZstdCompressor instance.

The threads argument controls how many threads to use for compression. The default is 0 which means to use a single thread. Negative values use the number of logical CPUs in the machine.

The function returns a BufferWithSegmentsCollection. This type represents N discrete memory allocations, eaching holding 1 or more compressed frames.

Output data is written to shared memory buffers. This means that unlike regular Python objects, a reference to any object within the collection keeps the shared buffer and therefore memory backing it alive. This can have undesirable effects on process memory usage.

The API and behavior of this function is experimental and will likely change. Known deficiencies include:

• If asked to use multiple threads, it will always spawn that many threads, even if the input is too small to use them. It should automatically lower the thread count when the extra threads would just add overhead.

• The buffer allocation strategy is fixed. There is room to make it dynamic, perhaps even to allow one output buffer per input, facilitating a variation of the API to return a list without the adverse effects of shared memory buffers.

Utility Methods

memory_size() obtains the size of the underlying zstd decompression context, in bytes.:

dctx = zstd.ZstdDecompressor()
size = dctx.memory_size()

Simple API

decompress(data) can be used to decompress an entire compressed zstd frame in a single operation.:

dctx = zstd.ZstdDecompressor()
decompressed = dctx.decompress(data)

By default, decompress(data) will only work on data written with the content size encoded in its header (this is the default behavior of ZstdCompressor().compress() but may not be true for streaming compression). If compressed data without an embedded content size is seen, zstd.ZstdError will be raised.

If the compressed data doesn’t have its content size embedded within it, decompression can be attempted by specifying the max_output_size argument.:

dctx = zstd.ZstdDecompressor()
uncompressed = dctx.decompress(data, max_output_size=1048576)

Ideally, max_output_size will be identical to the decompressed output size.

If max_output_size is too small to hold the decompressed data, zstd.ZstdError will be raised.

If max_output_size is larger than the decompressed data, the allocated output buffer will be resized to only use the space required.

Please note that an allocation of the requested max_output_size will be performed every time the method is called. Setting to a very large value could result in a lot of work for the memory allocator and may result in MemoryError being raised if the allocation fails.

Stream Reader API

stream_reader(source) can be used to obtain an object conforming to the io.RawIOBase interface for reading decompressed output as a stream:

with open(path, 'rb') as fh:
    dctx = zstd.ZstdDecompressor()
    with dctx.stream_reader(fh) as reader:
        while True:
            chunk = reader.read(16384)
            if not chunk:
                break

            # Do something with decompressed chunk.

The stream can only be read within a context manager. When the context manager exits, the stream is closed and the underlying resource is released and future operations against the stream will fail.

The source argument to stream_reader() can be any object with a read(size) method or any object implementing the buffer protocol.

If the source is a stream, you can specify how large read() requests to that stream should be via the read_size argument. It defaults to zstandard.DECOMPRESSION_RECOMMENDED_INPUT_SIZE.:

with open(path, 'rb') as fh:
    dctx = zstd.ZstdDecompressor()
    # Will perform fh.read(8192) when obtaining data for the decompressor.
    with dctx.stream_reader(fh, read_size=8192) as reader:
        ...

The stream returned by stream_reader() is not writable.

The stream returned by stream_reader() is partially seekable. Absolute and relative positions (SEEK_SET and SEEK_CUR) forward of the current position are allowed. Offsets behind the current read position and offsets relative to the end of stream are not allowed and will raise ValueError if attempted.

tell() returns the number of decompressed bytes read so far.

Not all I/O methods are implemented. Notably missing is support for readline(), readlines(), and linewise iteration support. Support for these is planned for a future release.

Streaming Input API

stream_writer(fh) can be used to incrementally send compressed data to a decompressor.:

dctx = zstd.ZstdDecompressor()
with dctx.stream_writer(fh) as decompressor:
    decompressor.write(compressed_data)

This behaves similarly to zstd.ZstdCompressor: compressed data is written to the decompressor by calling write(data) and decompressed output is written to the output object by calling its write(data) method.

Calls to write() will return the number of bytes written to the output object. Not all inputs will result in bytes being written, so return values of 0 are possible.

The size of chunks being write() to the destination can be specified:

dctx = zstd.ZstdDecompressor()
with dctx.stream_writer(fh, write_size=16384) as decompressor:
    pass

You can see how much memory is being used by the decompressor:

dctx = zstd.ZstdDecompressor()
with dctx.stream_writer(fh) as decompressor:
    byte_size = decompressor.memory_size()

Streaming Output API

read_to_iter(fh) provides a mechanism to stream decompressed data out of a compressed source as an iterator of data chunks.:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_to_iter(fh):
    # Do something with original data.

read_to_iter() accepts an object with a read(size) method that will return compressed bytes or an object conforming to the buffer protocol that can expose its data as a contiguous range of bytes.

read_to_iter() returns an iterator whose elements are chunks of the decompressed data.

The size of requested read() from the source can be specified:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_to_iter(fh, read_size=16384):
    pass

It is also possible to skip leading bytes in the input data:

dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_to_iter(fh, skip_bytes=1):
    pass

Similarly to ZstdCompressor.read_to_iter(), the consumer of the iterator controls when data is decompressed. If the iterator isn’t consumed, decompression is put on hold.

When read_to_iter() is passed an object conforming to the buffer protocol, the behavior may seem similar to what occurs when the simple decompression API is used. However, this API works when the decompressed size is unknown. Furthermore, if feeding large inputs, the decompressor will work in chunks instead of performing a single operation.

Stream Copying API

copy_stream(ifh, ofh) can be used to copy data across 2 streams while performing decompression.:

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh)

e.g. to decompress a file to another file:

dctx = zstd.ZstdDecompressor()
with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:
    dctx.copy_stream(ifh, ofh)

The size of chunks being read() and write() from and to the streams can be specified:

dctx = zstd.ZstdDecompressor()
dctx.copy_stream(ifh, ofh, read_size=8192, write_size=16384)

Decompressor API

decompressobj() returns an object that exposes a decompress(data) method. Compressed data chunks are fed into decompress(data) and uncompressed output (or an empty bytes) is returned. Output from subsequent calls needs to be concatenated to reassemble the full decompressed byte sequence.

The purpose of decompressobj() is to provide an API-compatible interface with zlib.decompressobj and bz2.BZ2Decompressor. This allows callers to swap in different decompressor objects while using the same API.

Each object is single use: once an input frame is decoded, decompress() can no longer be called.

Here is how this API should be used:

dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj()
data = dobj.decompress(compressed_chunk_0)
data = dobj.decompress(compressed_chunk_1)

By default, calls to decompress() write output data in chunks of size DECOMPRESSION_RECOMMENDED_OUTPUT_SIZE. These chunks are concatenated before being returned to the caller. It is possible to define the size of these temporary chunks by passing write_size to decompressobj():

dctx = zstd.ZstdDecompressor()
dobj = dctx.decompressobj(write_size=1048576)

Batch Decompression API

(Experimental. Not yet supported in CFFI bindings.)

multi_decompress_to_buffer() performs decompression of multiple frames as a single operation and returns a BufferWithSegmentsCollection containing decompressed data for all inputs.

Compressed frames can be passed to the function as a BufferWithSegments, a BufferWithSegmentsCollection, or as a list containing objects that conform to the buffer protocol. For best performance, pass a BufferWithSegmentsCollection or a BufferWithSegments, as minimal input validation will be done for that type. If calling from Python (as opposed to C), constructing one of these instances may add overhead cancelling out the performance overhead of validation for list inputs.:

dctx = zstd.ZstdDecompressor()
results = dctx.multi_decompress_to_buffer([b'...', b'...'])

The decompressed size of each frame MUST be discoverable. It can either be embedded within the zstd frame (write_content_size=True argument to ZstdCompressor) or passed in via the decompressed_sizes argument.

The decompressed_sizes argument is an object conforming to the buffer protocol which holds an array of 64-bit unsigned integers in the machine’s native format defining the decompressed sizes of each frame. If this argument is passed, it avoids having to scan each frame for its decompressed size. This frame scanning can add noticeable overhead in some scenarios.:

frames = [...]
sizes = struct.pack('=QQQQ', len0, len1, len2, len3)

dctx = zstd.ZstdDecompressor()
results = dctx.multi_decompress_to_buffer(frames, decompressed_sizes=sizes)

The threads argument controls the number of threads to use to perform decompression operations. The default (0) or the value 1 means to use a single thread. Negative values use the number of logical CPUs in the machine.

This function is logically equivalent to performing dctx.decompress() on each input frame and returning the result.

This function exists to perform decompression on multiple frames as fast as possible by having as little overhead as possible. Since decompression is performed as a single operation and since the decompressed output is stored in a single buffer, extra memory allocations, Python objects, and Python function calls are avoided. This is ideal for scenarios where callers know up front that they need to access data for multiple frames, such as when delta chains are being used.

Currently, the implementation always spawns multiple threads when requested, even if the amount of work to do is small. In the future, it will be smarter about avoiding threads and their associated overhead when the amount of work to do is small.

Prefix Dictionary Chain Decompression

decompress_content_dict_chain(frames) performs decompression of a list of zstd frames produced using chained prefix dictionary compression. Such a list of frames is produced by compressing discrete inputs where each non-initial input is compressed with a prefix dictionary consisting of the content of the previous input.

For example, say you have the following inputs:

inputs = [b'input 1', b'input 2', b'input 3']

The zstd frame chain consists of:

1. b'input 1' compressed in standalone/discrete mode

2. b'input 2' compressed using b'input 1' as a prefix dictionary

3. b'input 3' compressed using b'input 2' as a prefix dictionary

Each zstd frame must have the content size written.

The following Python code can be used to produce a prefix dictionary chain:

def make_chain(inputs):
    frames = []

    # First frame is compressed in standalone/discrete mode.
    zctx = zstd.ZstdCompressor()
    frames.append(zctx.compress(inputs[0]))

    # Subsequent frames use the previous fulltext as a prefix dictionary
    for i, raw in enumerate(inputs[1:]):
        dict_data = zstd.ZstdCompressionDict(
            inputs[i], dict_type=zstd.DICT_TYPE_RAWCONTENT)
        zctx = zstd.ZstdCompressor(dict_data=dict_data)
        frames.append(zctx.compress(raw))

    return frames

decompress_content_dict_chain() returns the uncompressed data of the last element in the input chain.

Training Dictionaries

Unless using prefix dictionaries, dictionary data is produced by training on existing data:

dict_data = zstd.train_dictionary(size, samples)

This takes a target dictionary size and list of bytes instances and creates and returns a ZstdCompressionDict.

The dictionary training mechanism is known as cover. More details about it are available in the paper Effective Construction of Relative Lempel-Ziv Dictionaries (authors: Liao, Petri, Moffat, Wirth).

The cover algorithm takes parameters k` and ``d. These are the segment size and dmer size, respectively. The returned dictionary instance created by this function has k and d attributes containing the values for these parameters. If a ZstdCompressionDict is constructed from raw bytes data (a content-only dictionary), the k and d attributes will be 0.

The segment and dmer size parameters to the cover algorithm can either be specified manually or train_dictionary() can try multiple values and pick the best one, where best means the smallest compressed data size. This later mode is called optimization mode.

If none of k, d, steps, threads, level, notifications, or dict_id (basically anything from the underlying ZDICT_cover_params_t struct) are defined, optimization mode is used with default parameter values.

If steps or threads are defined, then optimization mode is engaged with explicit control over those parameters. Specifying threads=0 or threads=1 can be used to engage optimization mode if other parameters are not defined.

Otherwise, non-optimization mode is used with the parameters specified.

This function takes the following arguments:

dict_size

Target size in bytes of the dictionary to generate.

samples

A list of bytes holding samples the dictionary will be trained from.

k

Parameter to cover algorithm defining the segment size. A reasonable range is [16, 2048+].

d

Parameter to cover algorithm defining the dmer size. A reasonable range is [6, 16]. d must be less than or equal to k.

dict_id

Integer dictionary ID for the produced dictionary. Default is 0, which uses a random value.

steps

Number of steps through k values to perform when trying parameter variations.

threads

Number of threads to use when trying parameter variations. Default is 0, which means to use a single thread. A negative value can be specified to use as many threads as there are detected logical CPUs.

level

Integer target compression level when trying parameter variations.

notifications

Controls writing of informational messages to stderr. 0 (the default) means to write nothing. 1 writes errors. 2 writes progression info. 3 writes more details. And 4 writes all info.

BufferWithSegments

The BufferWithSegments type represents a memory buffer containing N discrete items of known lengths (segments). It is essentially a fixed size memory address and an array of 2-tuples of (offset, length) 64-bit unsigned native endian integers defining the byte offset and length of each segment within the buffer.

Instances behave like containers.

len() returns the number of segments within the instance.

o[index] or __getitem__ obtains a BufferSegment representing an individual segment within the backing buffer. That returned object references (not copies) memory. This means that iterating all objects doesn’t copy data within the buffer.

The .size attribute contains the total size in bytes of the backing buffer.

Instances conform to the buffer protocol. So a reference to the backing bytes can be obtained via memoryview(o). A copy of the backing bytes can also be obtained via .tobytes().

The .segments attribute exposes the array of (offset, length) for segments within the buffer. It is a BufferSegments type.

BufferSegment

The BufferSegment type represents a segment within a BufferWithSegments. It is essentially a reference to N bytes within a BufferWithSegments.

len() returns the length of the segment in bytes.

.offset contains the byte offset of this segment within its parent BufferWithSegments instance.

The object conforms to the buffer protocol. .tobytes() can be called to obtain a bytes instance with a copy of the backing bytes.

BufferSegments

This type represents an array of (offset, length) integers defining segments within a BufferWithSegments.

The array members are 64-bit unsigned integers using host/native bit order.

Instances conform to the buffer protocol.

BufferWithSegmentsCollection

The BufferWithSegmentsCollection type represents a virtual spanning view of multiple BufferWithSegments instances.

Instances are constructed from 1 or more BufferWithSegments instances. The resulting object behaves like an ordered sequence whose members are the segments within each BufferWithSegments.

len() returns the number of segments within all BufferWithSegments instances.

o[index] and __getitem__(index) return the BufferSegment at that offset as if all BufferWithSegments instances were a single entity.

If the object is composed of 2 BufferWithSegments instances with the first having 2 segments and the second have 3 segments, then b[0] and b[1] access segments in the first object and b[2], b[3], and b[4] access segments from the second.