compress_pickle

Standard python pickle, thinly wrapped with standard compression libraries

The standard pickle package provides an excellent default tool for serializing arbitrary python objects and storing them to disk. Standard python also includes broad set of data compression packages. compress_pickle provides an interface to the standard pickle.dump and pickle.load functions, but wraps them in order to direct the serialized data through one of the standard compression packages. This way you can seemlessly serialize data to disk in a compressed way.

compress_pickle is built and tested in python >= 3.5

Supported compression protocols:

• gzip
• bz2
• lzma
• zipfile (Note that python3.6 and higher allows to build a file-like buffer into the zip archive, which allows us to use less memory than in python3.5)

Please refer to the package's documentation for more information