Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.6.2-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b99157758b5d3ba11c46db26602750555053aee2b917ba3209eaf37ee266ccb4 |
|
MD5 | 90d10b2460aa24e7b4fda530a868b013 |
|
BLAKE2b-256 | d93eb4164fc2f68ecaf766133a9e9f0b32ffee983f1fe9dd540fba1e23737237 |
Hashes for blosc2-2.6.2-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3025e4d0bdab498853e0cf971ece10ac5709c875f0b6b4272fe069326b69ef42 |
|
MD5 | aa31c9dd629c919140e10a63ac654e2c |
|
BLAKE2b-256 | 0aaa4c07aea2c2735dfbe2098c6f3de4adafca096383166daf1eaebaafb122a9 |
Hashes for blosc2-2.6.2-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a63ad8ae52af974d4ffa1490aa7715cfe0d6408363686fb141ff6f7513bb0ad |
|
MD5 | 38ce56883e308d4ee568160b288f26cd |
|
BLAKE2b-256 | fad998aaf7c9445580ad8bc936739700c0619cd5401edb691604aeb7745109fa |
Hashes for blosc2-2.6.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c4fec36f58267fa0b5b1ed7f688469313e5af83ed1cc70ba01001d3fe4b824f |
|
MD5 | f6795fd354fea99915b9aac303b23f3e |
|
BLAKE2b-256 | f8786a5722135283264d5a52fc08f9f47bef96cc7478229d43fbd42ba77edb14 |
Hashes for blosc2-2.6.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ca4c24cb1f64dba1b900fbfc165649bbfd9c890d76e356a682a9cff4c34f967 |
|
MD5 | 66ba09cecdb8451962188a0b4c75eee3 |
|
BLAKE2b-256 | ae5101dbcb0b0af327b6a8eeb05f3d3e335233ba1d38d2c4ebb3f04b73b27705 |
Hashes for blosc2-2.6.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4c2272915e0f28cd10258393506cc31616317d94fed77b61617c98734588016 |
|
MD5 | b7d2bcd999982a901b8bf92d4223340e |
|
BLAKE2b-256 | 7c193de7d29076d3eedbd5060ed9bd551fa488dda438090fa30b7a671b15bca2 |
Hashes for blosc2-2.6.2-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb8fc5c0420eab9c4c0c7eddf1b8747b817f7aae5145e3e99607918af3f42588 |
|
MD5 | 84e3ff8b855b3f55cb47aa1ec7a16fcd |
|
BLAKE2b-256 | 64986f0e0e66f395c5123c74c7acc4626622e34fa1ddf2b45f0684d7a9c81bee |
Hashes for blosc2-2.6.2-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db38cc7aed6547f0855ef5dbb13853f653a91174bf5e79841dd00ff1914a83d3 |
|
MD5 | 88e355762d7b84df834511767749e2e3 |
|
BLAKE2b-256 | 916ce43be26626f5c2b3eaff83c2b652316e0884c9733191df74aac812ef91ea |
Hashes for blosc2-2.6.2-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c5b861a8c51af1cd7eabf59c3bdd944f873ea5de8497602af9c5617cabe4f7e |
|
MD5 | 6ce611e69b5b9cd4c9085a02ec85c66e |
|
BLAKE2b-256 | 0af42e3f0230eb961bd88c7f52bd091627f3c4bc3c590af78aa77c176cf84f54 |
Hashes for blosc2-2.6.2-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be1925fbf1ce37d384f47d3f02710abe79cc7722d09c2d50044845947e85d2fa |
|
MD5 | ce9a9abe0c595e45629ba189bc96ffc6 |
|
BLAKE2b-256 | 9a820634ae77bdfa53419b8c0a56dc0546f8eb8d5b14e34aab3f2600fd67de8f |
Hashes for blosc2-2.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81b0d7cf029b097fd130817ddae66e67a92253136812a5dddba3d9504bce15ed |
|
MD5 | 893d930b61f6eedee0e40a4d14275228 |
|
BLAKE2b-256 | 31ff1de7cc9606793bc85d234b46612925246927909adda96bd85bb42c39fbdb |
Hashes for blosc2-2.6.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50661d5e9147b8f50a86c7d86ec2be907ac33418c5ec82963f4487d851e9c88c |
|
MD5 | 8bb5b643ba44715a9a820b255206d274 |
|
BLAKE2b-256 | 7747dcca6bbe796f386fbbd16520c57feccfa672ea6439e4848655138ff69610 |
Hashes for blosc2-2.6.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cbe8e97f0bc94a45456f186c374e5fb91d35123ebe80e530d849d1da95cf6770 |
|
MD5 | 0f22a240a739013deabb25f47c7ba20a |
|
BLAKE2b-256 | 2440f81eb3a8bbcc95aa7ff82df21f9d90a6819ee718fcecdf2ac6335ddbd401 |
Hashes for blosc2-2.6.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15d91ba9fd24391a67dcb1051b82490b0cbde3a1d473209fa578e7a96d801bf7 |
|
MD5 | 6eab8a79d1aa94f7bdbc95b20462ae34 |
|
BLAKE2b-256 | b8c01b211ee0cdf2b57f6edf8ad5edd5e198d215d4828fe8a8cb7534c89fa425 |
Hashes for blosc2-2.6.2-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8291cd97f3730873c498df610acb0177ff11901e09771197e1eace5c3e1b9669 |
|
MD5 | b86a50ade36cf4aa0a7a49ee04488152 |
|
BLAKE2b-256 | 207326ac0a44dc7668c5862728a46cf2b9b51ab69cd1716ca348f4a11f79d325 |
Hashes for blosc2-2.6.2-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | abc87b8bda70290a33b0d5631121d189f90046b86f7992865428672471cccba0 |
|
MD5 | 71cd6694690290e76a1fe8249d8dab39 |
|
BLAKE2b-256 | 0b7b353fdacd5efcc96e9bf4b7374e24a050b2ab6cfd77439060c3d020c67f7f |
Hashes for blosc2-2.6.2-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afb962aef4f2b3b5cd20a3ae2d92311bb829836ae283b5ac595fa14dd2fad47c |
|
MD5 | 5fbec00f6e6b726e096732f05d8931d8 |
|
BLAKE2b-256 | 7d8b1de981ee44406387b49e3609588d5743913ad32ca1cfa35a8f7fe27d5551 |
Hashes for blosc2-2.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 072a8753d627499893129d480042a61ee47845ce99106fa0e7d8ea4f0ced37a1 |
|
MD5 | 41563ca023ef9422bbf72a932a9b42ff |
|
BLAKE2b-256 | 46836daa9f0d8c5164c01a61f736175fd79def0141aace3c12446ec0f9ebf4c4 |
Hashes for blosc2-2.6.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8aeb8eb12c60522bf0eb6d49687aba925e710ba4f9976cdde519d7af3bc547df |
|
MD5 | 40b4d728c49d2e14d6109efade2de665 |
|
BLAKE2b-256 | 660b6593d4c6cc3be08fb02bfa540197300db822ee8cf19b393d3e1e17bd75b3 |
Hashes for blosc2-2.6.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 407627050d116d1cce85b197616350d3f2852f7e036a4f59a97d5cc07f345ead |
|
MD5 | 3b539dad31afb65646f93ca5aa25b131 |
|
BLAKE2b-256 | 4c80a13e60558100f0617b5c14ac86222b1f162be7ad3b9b9fa67be4ec792a1c |
Hashes for blosc2-2.6.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00db67006601f534553a7948213595f384eac0e3afa41a4f5600fbb3ba580ae2 |
|
MD5 | 6fccc467605f000819b60d9eab4ff33f |
|
BLAKE2b-256 | c8a76056dd3c5a22dc1cc4e4eb95203399215c4c9d27e29716e86696305ed631 |