Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.6.1-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cabf6d9d96f25741d78348d1485dbdae963f98a464e953fbf645a103f7742e8a |
|
MD5 | 8ff1f50d911d74f39af390886d048a03 |
|
BLAKE2b-256 | 4c6c3002c279a7f00aa9fe60da0375fe337b56f4287855db9f63ad0f2678b9db |
Hashes for blosc2-2.6.1-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7d0b0db44d54ee90889d4f9b2e76892ad0437d87bf01602d254c8912886e47a |
|
MD5 | 377f29bd69a3767a9081168cb410e5c2 |
|
BLAKE2b-256 | ad03e35bb80078682088790b67ba47d377d4e67da0c2a25d6102a5f0b48a9de8 |
Hashes for blosc2-2.6.1-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28286e8977e5b92068df181db69546e4a31269b01794df1d3f6b9ee592bf0c2f |
|
MD5 | ecc8f9c11a9af31764d7b4fe16d105fc |
|
BLAKE2b-256 | e21a381729dddd4967c9fc5ba841adc8f53471be64f09ee33eb075bf52de7fbe |
Hashes for blosc2-2.6.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | decdaba2ef3fbfcf6d1c276c8c9de37bbe0275daefc3bb27a5234800bcb77c8d |
|
MD5 | b19c0602e18b354b953f5f11ba2dd376 |
|
BLAKE2b-256 | 8687905eaf7ba97d8a53f61005d4a03ccb0708bd4409e1db44a18421679ecbdc |
Hashes for blosc2-2.6.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f251ffcf12db2f27428512223c63a40e881a659c72d729bd22ad2ce2f4f38df |
|
MD5 | e4aa98d142cdd07b31297f59f3683656 |
|
BLAKE2b-256 | 6a2e775c3cfc705f419f8fcbaaf3ab49661a535e320f00dbb609dff14a4b12e5 |
Hashes for blosc2-2.6.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09f31a31237dfc58fb6f05d61dcf5f8ead5bd84cb3ada7a2b64adab62b4ce53a |
|
MD5 | bb3f853ba395f31be99111a22162d26d |
|
BLAKE2b-256 | 13f11c01ee8b3a49e76c895632fdb2487ca57c274c304afb5832c59c15c765c5 |
Hashes for blosc2-2.6.1-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e2bf289a867b346d240838eb4a8bd83de759be2b719c5ec581b7e4bcbb6c99d |
|
MD5 | d81e107769bbac85f7611e1e92308a39 |
|
BLAKE2b-256 | 3fccce7fb1ade08c1244e2de316de8ba6cc5637c4359b1f7a389fc2d7acc8d3f |
Hashes for blosc2-2.6.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c23087b13693cb6a55437dfb8c025752f5eb0a1d63a65f4a15d5e8bb416d303b |
|
MD5 | abce7fe34da7c505694603aedf0872e6 |
|
BLAKE2b-256 | 8af7f65821842430e8cae728bbe56563463f1e408a229ad31878b1047ab0625c |
Hashes for blosc2-2.6.1-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30eb55008b45d1e4047143fd0f4e1866993d59c03eb11c9966ec25ca3973bd02 |
|
MD5 | f64d7b4fb2d23ce0f10b0a90e872e300 |
|
BLAKE2b-256 | 566f85696e30fa93ba7a0ccd396023f4930ec79d7d1a59e39b18392311bbed4b |
Hashes for blosc2-2.6.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df38c013b26cf0f6c5a6cb344420bb496ba0ce99f2fe3fac5aecfc9fc5969d91 |
|
MD5 | e4febfb5c0b92d59738d86bf0d91d3df |
|
BLAKE2b-256 | 1a9194c4efa85e9564b6d86dfec3a65daf56f7ef3847ecf6ba16270f7ffd1576 |
Hashes for blosc2-2.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95d8dcb1c6e565ebda1119e60c4a87a3cd6a49c821bef519fb88581eecb12a15 |
|
MD5 | c64ddaad86fc98fc12ef5887eb05ac17 |
|
BLAKE2b-256 | 6bb52c15478d691081aabded334a765b92abdb806a8d994250ea236e49b7b377 |
Hashes for blosc2-2.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e06eea3f81ef9ac15490b97f8153601c7baf9bdc6b31968f2312650ec5b2e31 |
|
MD5 | 02c8954a64c7eabe74a862ccdc62b4a4 |
|
BLAKE2b-256 | bbbb6f1fa41bdb5a6a821b9b3ad28e01a4c6713626696f6ae727f0146b72d824 |
Hashes for blosc2-2.6.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1f55567884b6c24da1c39985109266253579e432c2530994765e9e6e1fd9011 |
|
MD5 | 4d3ed02550df83f1ec2a0c5f50e0bcb0 |
|
BLAKE2b-256 | dcae124b3d3878ac62e60cc97c150c0d4c6f4867caa42170ed1e602ede132841 |
Hashes for blosc2-2.6.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5de94c6d05caf4df69e215d4522d1645aacecad05deeb34293f21fb0017f3454 |
|
MD5 | 6f9a28d0c194766cbbdc1f1cb7f9e791 |
|
BLAKE2b-256 | 2c51d1505843689b2bd956e30a0556c7f716c4d0e822691a86d5a5f22b42acdd |
Hashes for blosc2-2.6.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bbd23c793c5ceee5b0ce4a2f77e0cf7fc095bc1b2071ce271278524f96ceee7 |
|
MD5 | 6c1ef27552260761911fdbd62d4eaa4e |
|
BLAKE2b-256 | de9eed12a8c33599148370c68bdead54f318e8f763fcc7dd6296e301bdc782cf |
Hashes for blosc2-2.6.1-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38552c52cd043962e6d2990dc7a398d4f946c1cb9c009e5cb0b4a9601e896e95 |
|
MD5 | 0800a03312ca44578fb31c3cd9a4c558 |
|
BLAKE2b-256 | 8e47d1da96f8359405c264b0b38f9eae18c6f87ae10d342df5ae4b13e97fe201 |
Hashes for blosc2-2.6.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6959271f336059ad6d3164e0c6643ba05c4753d42a119c42ddb68d06c26a3e7 |
|
MD5 | df30fede055f8ab88fad2f4b4edb0c29 |
|
BLAKE2b-256 | 3a02457b1974b87d9d07a506ffbb6033e31ae9721d4e5edeb5fc99f7f1d7fdbf |
Hashes for blosc2-2.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a476534baafe332db4997bcf53b5c83716d857af05ab1abace24a5c69c968b6 |
|
MD5 | 5f5620ad1bd9c17819c7a832c639de5f |
|
BLAKE2b-256 | c896c33195ceab967a295673101f0ef06faf78aadfd5830ac7557906d13a0132 |
Hashes for blosc2-2.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac1af499394ce852d66e9f819d8333fbe52e4bb7a6010d744cf68b912fc9e91c |
|
MD5 | e449fce13a5a0e15b0a01ea9ef9498b9 |
|
BLAKE2b-256 | 30ba01a8f99834ca3b1cd79c9a0430565652f8c944bbae8e3fba770a141bf802 |
Hashes for blosc2-2.6.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebb85bd1b0864d5abe53fef3319306b9a89231d5295a31c50d845bba37c985c1 |
|
MD5 | 64c9cc7290dde84d5c9d0b6369a92bf5 |
|
BLAKE2b-256 | 34c226a9e0e8856dccaaa935b8cf5641a2b8e5a2b56c490cc8e040fe1c1b8738 |
Hashes for blosc2-2.6.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4be4398b2f39a336958e1e1ce8aa59923d1137ebdd46dd7821390e725168300 |
|
MD5 | 0035e2c61e6c2d668b46687c7d2579e6 |
|
BLAKE2b-256 | aa4677a86f29be79f56e098f4dcb2d57fe9fc7048445b4c59c7e91a720393138 |