Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.5-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78eca62700347452d5d4f7c0f9a789811d17a35709c0b181d93eb4f44bb93600 |
|
MD5 | 1e167b3236b16ad765c8974239273dbd |
|
BLAKE2b-256 | dc516746898e109cc730bce62f5c600b9bc1d250cdd7f2a5f2a7b4211f5a2f6f |
Hashes for blosc2-2.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cea84fa5ba603896b522afd782e42cf8adbb93dfcb4ff5349e1ba2f114794676 |
|
MD5 | 0fcd25e83c2467d995f8a7a89998a27f |
|
BLAKE2b-256 | 8b2967231597205620f932cfe3d1b09f279de5229f16c0343f4ecd2add4edb8c |
Hashes for blosc2-2.2.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a786d7984e51a132fa88ac1a2a429585a37bf4d2cf5e895cfc9013e5eb3ef39 |
|
MD5 | 62ab44cf535f8f0a638db5d8f7efdb93 |
|
BLAKE2b-256 | 91c9c68454664d278668c274503f327e2cc93c16451ecb3022cc9eb0cef1f3e9 |
Hashes for blosc2-2.2.5-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d8a481a018241395f30c96745047448c51d36b9437f00f7c1913dfecd586443 |
|
MD5 | f66829be12a3ed6f22a28a8f3d8e1bb3 |
|
BLAKE2b-256 | 34cfec87aaa48687cbfc1f09bc3aed051ff846e81170cabd19c8456f6cb4ea3a |
Hashes for blosc2-2.2.5-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4715fab746a7cd3815d3117805e43c58e84f732db376d698245fd1181ff68eae |
|
MD5 | b1b05dbe42bcb69d2516abb7673f53b0 |
|
BLAKE2b-256 | 4ba9d1b8651a0c912f273351c2528270502a343440c6e3d300f92f42379783cd |
Hashes for blosc2-2.2.5-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d91a4a6fdad9d93b958440231255874fccbeb346faeaf6d0468f308e1025024f |
|
MD5 | 479c6cf0a44783aae5fa0a2baca239ea |
|
BLAKE2b-256 | 34f63015302e4156217c6d4197c21617ca2c830dfe28a229280590a75a468622 |
Hashes for blosc2-2.2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | affcbdcd88d1889bdf633831f17c7a47700fc6030416b562bbb00a6e0f075d39 |
|
MD5 | 25031233ae675bb178a2799f6733042d |
|
BLAKE2b-256 | 4104c646249d6040ab03cc2b9d15f0406f401b9b06b23ca51e846acc08cb5854 |
Hashes for blosc2-2.2.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 99277cf357f2a6ae0253d44de40bc5aa38f5ce643e758a54b41ead12df94a633 |
|
MD5 | 2c7aa5ad1a2749510efc78feaab813f9 |
|
BLAKE2b-256 | b8d0c1d7a78865964d4885c3fd4be66bbefe34af2dd0ebcef45801c5685c2a4d |
Hashes for blosc2-2.2.5-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeed99519e1bf62d8de565b27f1a4ec12ced6fe7d0b2c632d8e8f4991577529f |
|
MD5 | a60cb1000c891b7f7961b65d3acfddd6 |
|
BLAKE2b-256 | 81fd5717d838a635654bd12f7514e1a8a3dffa55ba27a51fe223f69738c4699c |
Hashes for blosc2-2.2.5-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdfc93016944d1902d725c2e2615850c0ea3d4fca86497c77a64974e3280b5a2 |
|
MD5 | 5a29f20733ed8117c6135b22a4ae2084 |
|
BLAKE2b-256 | 92d242a6411b12ac1988554f671b351b42afa75b6be3adf256335adbf7f88471 |
Hashes for blosc2-2.2.5-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c09665f43545716bab222fcf3b8e763d116dc7002241ba0ddaeb83bb4d14642 |
|
MD5 | 34faf261e183fb2e0f7906624a1db448 |
|
BLAKE2b-256 | 6666ce24d402d9a2e8529eba7664f08387abb8fbc80fc421ef47c15b28b5a534 |
Hashes for blosc2-2.2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3e64f316309c451d96d3e4838a94b74bd956c87aa8407af4b7d63495f34722e |
|
MD5 | 7afc73a937fb52b0a3153d7bd58656ca |
|
BLAKE2b-256 | 2b80124ba450cdefe573fb033da3b2c9d4b48ca39ce2334a0e5642af935d1dc5 |
Hashes for blosc2-2.2.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee3a18b844bee29f67f0390d284d4aaa09128f35aa373a73c34542f685a35a52 |
|
MD5 | 0c9de87132be0b6caa64b9835dc3e753 |
|
BLAKE2b-256 | d0f0fafc4fdda2a21578cf3930888bd1a6ebc602a739fce98d66d129b336e2fb |
Hashes for blosc2-2.2.5-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a99374880bc95fad272cd5362727a4c7ebff7fdce154aa403c82afa88653a25 |
|
MD5 | 7030f03adf418c4fbeb3eae8a1063297 |
|
BLAKE2b-256 | b5b56c5cb4df313ed5de8d7ab9bb9bd332f873fd9ec87fdda4156785a1839422 |
Hashes for blosc2-2.2.5-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dbca224653ca0be5a9b245d1cf39d157809bc22c0ba9e1bcc93a4199fc3ca27 |
|
MD5 | 67f0d7db407e62ee5f882163d1f852ae |
|
BLAKE2b-256 | 91ebae093d90cde453702b7c1feb54a51639f8a908055b79fc01e478d0f01a67 |
Hashes for blosc2-2.2.5-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5eb8e17c776a4bc51bc06f5f01d35bfc2489115d0a8b78223a265370694f6295 |
|
MD5 | f3e92007a1c5800e522fade462801ce5 |
|
BLAKE2b-256 | bcc6c0499df033b044340e91df7c0678a79490832b9a357f9f57e621fb6a2d23 |
Hashes for blosc2-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21a03fdb092089c64dbb5d080b9f3b1ab2b769a7a46dc6e4bdf580dc95d3c268 |
|
MD5 | 8a87f3b11c828c930cfcaa34a424feee |
|
BLAKE2b-256 | 5d003732e0a38552d75819d5aab3551ea1aa5c2c3078447c485358538e952aef |
Hashes for blosc2-2.2.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 270a465794b3d44abcac12bd4d3883a43846c782f9d2672e533df88ffa0f655a |
|
MD5 | 70c21b6abc3bfc4e74106268a4d8c1c6 |
|
BLAKE2b-256 | e8194627fcdb3a9fd11da7b9dddd489223eb197820a854f016d0dddcad1d45e3 |
Hashes for blosc2-2.2.5-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80cb1d62196cc0dfac873f763e7102a9cd0f18f5ea190e072a429d19eb3ea679 |
|
MD5 | 8fa868e58ec9f1c8dca422ac8d8e06e1 |
|
BLAKE2b-256 | 444256e3811e47f3fc16e5a24e3aa99011d8a80059b3eaf38912062e48daebc5 |
Hashes for blosc2-2.2.5-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f4882e97f810d8f30918140275cc1b38f204c54778ba5a41e94f1145d998482 |
|
MD5 | 90600d595bea0567a6b70a6324372d7d |
|
BLAKE2b-256 | 6aa1ad148fde02d1b60aec0f09d81aea8e7fe5ff34513a94a31dc24b62337083 |