compress_pickle

Standard python pickle, thinly wrapped with standard compression libraries

The standard pickle package provides an excellent default tool for serializing arbitrary python objects and storing them to disk. Standard python also includes broad set of data compression packages. compress_pickle provides an interface to the standard pickle.dump, pickle.load, pickle.dumps and pickle.loads functions, but wraps them in order to direct the serialized data through one of the standard compression packages. This way you can seemlessly serialize data to disk or to any file-like object in a compressed way.

compress_pickle is built and tested in python >= 3.5

Supported compression protocols:

• gzip
• bz2
• lzma
• zipfile (Note that python3.6 and higher allows to build a file-like buffer into the zip archive, which allows us to use less memory than in python3.5)

Furthermore, compress_pickle supports the lz4 compression protocol, that isn't part of the standard python compression packages. This is provided as an optional extra requirement that can be installed as:

pip install compress_pickle[lz4]

Please refer to the package's documentation for more information