Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.7-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb1b6b6a46e3246e55eb69b438a61fca2428db3bbac3b0d5335b23bae7bfdaeb |
|
MD5 | 75e9f3c46528658033786d7ec63c7c68 |
|
BLAKE2b-256 | e4a08e04e2a2da84af69b7c7a8676eeb67a01a9792c19996f28a30dde63566c4 |
Hashes for blosc2-2.2.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc21e1024856160e14196b9dc61dc3f8a38bf6572ed0abb656f83d321359d130 |
|
MD5 | b11477b4e6b408b1f0ac76eba549acd3 |
|
BLAKE2b-256 | d3fc694d352e8bb4d66ff2b69390fb7fce1d4d647a1b3d9e696c4d1a55339951 |
Hashes for blosc2-2.2.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e69e7556bab20342db30e9fdfd7c198b12c8c5306c4b4831702b62efcec5298f |
|
MD5 | aab9991d5dd374d2b2e8f36572d69527 |
|
BLAKE2b-256 | 9d1e6e0c01d76aa2b5a7a2e8dc63ae423f8427765198affe182abc26b64d44f5 |
Hashes for blosc2-2.2.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 676d38201087aaae953701e2006d331c3232a7b091a78077d9c3a794a51c10ff |
|
MD5 | ee6bf67768b04b5c5b5221c0be0b777f |
|
BLAKE2b-256 | 393219508648443c7d97863f5df2f7dd900a45558a7863b3ead227d0e6091abe |
Hashes for blosc2-2.2.7-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76683b4da807cb8e5add36f9098f89bfc8e928758e1a5ae2319e952cc014bd08 |
|
MD5 | 41dceeffa1d8c717b247b1673d121ec8 |
|
BLAKE2b-256 | 5b0f0161b82665eb8d450943a7476bad30518048f0a3518f25e736e5727b39d4 |
Hashes for blosc2-2.2.7-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25effec18d3a2a30dfc6b828735141a806be049d6979aeae34e922ab42ecc288 |
|
MD5 | 8a9ed85518a474183747bacd4e49fc36 |
|
BLAKE2b-256 | 50167e7e01ff544079aa85e102be30f1b04e6c49497b02ef5ff47a628f02f5b2 |
Hashes for blosc2-2.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 175cb2ea313cd1676952519cf33fbc9931356cb0b4df61894c82a152aaffc6da |
|
MD5 | 7b6d4aeb67b5dbc3054ae83cc9fc80d0 |
|
BLAKE2b-256 | 425cb4258c15c7a5333a67b9601368235b6cf2aac1cbff40cbc5f707d34af611 |
Hashes for blosc2-2.2.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab5e43ea61794c59af072aac4adeed4bef9806881c94cb3181da72870ef77528 |
|
MD5 | 533b026ee4669f51cf603d19eda6f448 |
|
BLAKE2b-256 | ac1f2d34a917cdd4e1aa4ba068cc533c1da134b462ac5f20caaec8008f113974 |
Hashes for blosc2-2.2.7-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73c635be41b0a04e290c85b2c3b1a06bd4af7bbf9556686f0b00c7306e3a376f |
|
MD5 | 9a64f8293c967dcfb501bb24350a7d73 |
|
BLAKE2b-256 | e9f91d6f0d2dc70499e8eb7a389fea31898c5c834296412c5a17e083fc58bb99 |
Hashes for blosc2-2.2.7-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b089c24831f23b49e491fd3c57d8b4ca2921739b55816b68905fd1e5d173334b |
|
MD5 | 8ea29540b6408264033d467051f48160 |
|
BLAKE2b-256 | 247752fb923c9e78edd42132cfd67a4f639491efe3c87032c552d6962fa91313 |
Hashes for blosc2-2.2.7-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05c7eafd31598072797b92f7de67ae575b57caa5879b75d699e8c138517aee22 |
|
MD5 | 988bac8577ff2ecd337dc0e4f63e5af7 |
|
BLAKE2b-256 | ebaeb1683ff2af362b7ac7298861526ff38f0d663243ae7650029b4844a5e33d |
Hashes for blosc2-2.2.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be0e0e5bf5fa458af42aaa76deb5e901da751ead215448c2b3d2f046005cf88d |
|
MD5 | 283d3461f567bb73d6a199e57430c96d |
|
BLAKE2b-256 | 5410afe42d81959a2d8843481f2e20caca378af08ca4629e129c608bd5120d49 |
Hashes for blosc2-2.2.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8cdc1017a7f97769b7c4cfb772294aba0fdeb894efa8bf31d68b0ad2b859169b |
|
MD5 | d83aef727da25b96e649e6b4e7117835 |
|
BLAKE2b-256 | bce8a92d462219284ddf767362c710b8cc4a52b351b29fbbf6ebab95e4e10355 |
Hashes for blosc2-2.2.7-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9498db576a402b77cd5e89e56cb58c7f45bc20eb0f482612cdfc3025baa31961 |
|
MD5 | bb9a9316c1f0d23b612409bc7eace43a |
|
BLAKE2b-256 | a3dd968ea5f3c5f9cc552b7734416e3a4318620c1da1de874c796d2202147f1e |
Hashes for blosc2-2.2.7-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85c76f7025a325831f0610c2848cee46a88fe5dc89813e62d00b3bc8a28c3fed |
|
MD5 | 16986b7a9bea390bde437fc9949bcca5 |
|
BLAKE2b-256 | 0a4840426df65eaf48285404230e22b3baa701d2c89bbb6aa1248fc74c7f2fa1 |
Hashes for blosc2-2.2.7-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e165221bd7982786154f7b6acc30dd6b343ec4d8d15a66871306dd14acfa835 |
|
MD5 | 3852ccd0ffe2d2beed9a33b92ee2a771 |
|
BLAKE2b-256 | 23be2e71b7f66fcb3e287bf0d3863bb37b5a59e12b3b97b4cdbdff292785e0a7 |
Hashes for blosc2-2.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32417cd414e422257fefbc1846ee7f5293cf69a88f3210ff365d296cb806b941 |
|
MD5 | 821e13ede7837a1ee24df104fee12c44 |
|
BLAKE2b-256 | 037089c0ff5c44ac10eccf7a0257810e08dbee6e4d5956ce547ceec1398519cf |
Hashes for blosc2-2.2.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 771215bd16f333624d99a2deeaca4d9f4d8be55b26f1dc6ca71e8c3795d2da07 |
|
MD5 | a7e3e4bee046c32326c9558e3bebb323 |
|
BLAKE2b-256 | 4a2d0525f1b7b3bd3bac31bc245cbfe3626e2b2cad26bb93e4876a53f52c70ca |
Hashes for blosc2-2.2.7-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f030789157b95bccb48a4ccd6e3f356ba529fb3470f39d62378ce6bbf6a165b |
|
MD5 | e2f9784b5230120204f9223d10ec567a |
|
BLAKE2b-256 | f9781c2330199b2997ed8839c1566301931f0243163d9ca8ddd81500d40e8428 |
Hashes for blosc2-2.2.7-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95234dc7c295b077e7e2e9324ca43c319c847dc453ec7d873d61a92d65568b37 |
|
MD5 | e4008cd8d2b1b38e9834ab7020233a6b |
|
BLAKE2b-256 | ca169ccaf5398f642e3b929adc96888a259ebc2d4628e35c5dae0974b919ed1f |