Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.6.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89b7583c4b9dfb896895adc605bebf1ff04a80455f7dabc48b826c2e1fb268b1 |
|
MD5 | e6a980322cbb30b59791a50c472842ec |
|
BLAKE2b-256 | 09d2167f4ba358be11def46227c2a1b39c9025a5d543d9f7d557f337d7508164 |
Hashes for blosc2-2.6.0-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e9a8f52fe9010c8ff5f315096dd6533fb3d1c14ece9b8590b3439aef8f729e1 |
|
MD5 | 8c37b5a030a0925931774b934715dcda |
|
BLAKE2b-256 | 6e54eae6eca4f7a0d72311481ea9f33a6ae054db9cebd5f2b5033bb347f90667 |
Hashes for blosc2-2.6.0-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7ac178828f861f5a44257ec7a9299ae20039fdd38ada4e742f216a7680f59c8 |
|
MD5 | 2518968cdc8485dc1c915581e56b1210 |
|
BLAKE2b-256 | 16f42a9110b6da50b459f6332e4d69c4a749159e5bd3fde6aa7fca5d4119cc35 |
Hashes for blosc2-2.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 097976a68c26a826fd2410eece0809f7949f5d1e3625176de38dc04a0bf6dc0d |
|
MD5 | 2796d1bb4ddf62fcf1f07c8623861a23 |
|
BLAKE2b-256 | dfe5307547ddd2b7d129433941005a7e8960cf02b3979b9f817f369bc8d0bf0b |
Hashes for blosc2-2.6.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc8152256940f89d98704efab46a1c002ccc7c6178f78b92977302f7eed06e39 |
|
MD5 | a027a088102c1ff4e4d6262c77834c10 |
|
BLAKE2b-256 | 2976190ec473dd1a8d8d559d4770919bcadda65431cce392ce7a01472d130cae |
Hashes for blosc2-2.6.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0304f3145f3c4e5992c2c683dd66e688172ce6544b338584a4369a38eff4093c |
|
MD5 | cfee94ab3e1649dc8153eeef23feb9bd |
|
BLAKE2b-256 | 1925d2950b24817f9604e601988a89dfa873114ea54695e85129a225f0daefe2 |
Hashes for blosc2-2.6.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6747bfb5ccafa47ef99243fda220be508b7b8324008b696046cf035d23020fe9 |
|
MD5 | b8e7324be2d6d8e47edbd6840a3b17b8 |
|
BLAKE2b-256 | 0c41b2ade0322ed5baec8b968fe580847e1acf82bb43ff77c83ab46810abf8da |
Hashes for blosc2-2.6.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9672cfbe751c7a8705b3660721d16c4c0ea857b3201a2872087c9a2bb9f87b3 |
|
MD5 | a884cdb69ce0ba6a5ce5a2ab00c71607 |
|
BLAKE2b-256 | 6e7a76f099465b07437eeae3c64a82c3273301bfd9af8d33724c52ffeb16c9f7 |
Hashes for blosc2-2.6.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9639629bd5a0accf4b5ac8da11985aebee5fb20def78ce96037fd29a220c82c |
|
MD5 | 8dfbe9a3330cde163dea773379392dc2 |
|
BLAKE2b-256 | 01027f9e3c6ea80b1a275c764dacd7229c5f8cfb3efecd78410e641f902e2f6a |
Hashes for blosc2-2.6.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e9a2fbd36781573779b7c1d7f493bc994f4fee085d4e88a2efc3d372f03f4ce |
|
MD5 | cfabb426e10e1fb4c9b5cdfbddaa140b |
|
BLAKE2b-256 | c8e0d7487e71f478bf0ea68ab2ac8710e36eabfe1b223b62289d7b2d8e08db5d |
Hashes for blosc2-2.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 960a6de48d9bf194b6458a1a59cc5f2d8e1d8e75415148a2a81d89e83593fe59 |
|
MD5 | 676355fe275bccb36817ae137dc87b95 |
|
BLAKE2b-256 | f19a7fc548a6b88e1919608e58885aa5485e972791c58a8392bad5b3bdf23f66 |
Hashes for blosc2-2.6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d44812ec44fd35338da5bb72ce93f79484395f90805f36591dd1523f645fa0f |
|
MD5 | 00ec1f4119e6bb58cdc7fdc7166c904e |
|
BLAKE2b-256 | 57e8ae88f05e60fb3eee9946a592a28c201e6f4ac790bb0506c21b48b2a6db48 |
Hashes for blosc2-2.6.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df7a961876ce7650d26dc66ae52567608ea9bd910746dcc44f3254d8b1bb5cc6 |
|
MD5 | ca0bcba63b8b42abb8b6651c60e16fa4 |
|
BLAKE2b-256 | cd3f44b68af60824fe4d921ad1f1255a92f5335c46c87529a84d1f4cc105e1a1 |
Hashes for blosc2-2.6.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 633f9dcd3b4712b1dc38a77f631fe3d8e712bc1caa1a896ef861de8fd1b07aef |
|
MD5 | c421e5ee99ab017aa711002a1d853a1b |
|
BLAKE2b-256 | 6343eca77b36c4f0e7bd947586776971b6ca58d862f7cdc174c48dc542f0fe4c |
Hashes for blosc2-2.6.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89a092093ffe0b6fd535a76707c5c58cc860aca582a42cf110a8ca19c1f7ada2 |
|
MD5 | 6b817779c583e2031787178f65d1d5ca |
|
BLAKE2b-256 | 3024727a1deb09e5095a42b171449bfd44a4a2dbe04c0f37e95b3ff5ad2094dc |
Hashes for blosc2-2.6.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7bd3ff65e8105081bf980bf830b562ea6260c1b1e12810007e513f35f80f74e |
|
MD5 | 06d6954b154b4f957ebb22daa29898cf |
|
BLAKE2b-256 | f977a5f25065771603c6e6acc3fd0491cc50b1077784df66b17d7f931af77924 |
Hashes for blosc2-2.6.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b081d5fae153f9d1ef2864155d2a49d287cc404c2a574d66c3ce06b238f48320 |
|
MD5 | 4155080d7185ed1c3334453d612ef3c3 |
|
BLAKE2b-256 | 7c9f7e072236bbea74e3e0d42980dd2b0eca5af8e02e321ac3945fa5a3b2cd81 |
Hashes for blosc2-2.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4be2bc0491c8aa84aa6afe9bd031712e61f384ba202ee8e8bfd79239c754e915 |
|
MD5 | 528f8ff7863242a47af83af31aedb8db |
|
BLAKE2b-256 | aa91ef7e0763d09bbd50822fe303aa0fc827eb5b5befd48199509ac9efc4843e |
Hashes for blosc2-2.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e09a756c5692802d063bf84e12891d4dd61bbb6493141596b3ad2a031fbf427 |
|
MD5 | e15ac3edf38cb7b3beb8434368e8018a |
|
BLAKE2b-256 | 689c056e8fecbd570bd17d32e4ded8705a9c68037b9453e8d8b7b12cfd2f83cc |
Hashes for blosc2-2.6.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4142591a6bd6653961fa1f1f0f822150dd9464520f34403ae24b18201eaff324 |
|
MD5 | 2bc880c11a5f0bf033378744b6d33866 |
|
BLAKE2b-256 | 31cf0fb7d1e2aa791edbb4214fbda8f249c9f9fff0e3cee16dcdd0a9e4426e72 |
Hashes for blosc2-2.6.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc5c2da7564b6c3c5ed829cba876b586b0072ead5cdebd629ab33d3e5f4aa78a |
|
MD5 | 846e984713ca5558f85a84f5d038e125 |
|
BLAKE2b-256 | f046774acdf5e11c50fc5f88e90c0e4171de3d80d29b39fd279333a0574014a7 |