Python wrapper for the C-Blosc2 library
Reason this release was yanked:
Not working because AVX512
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.3.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c30836a3bd2de31577e5f82ed77d0ce07bf7ce95da59cfb7c160bad1da79a86 |
|
MD5 | cabcae51f4e2a4c6ed2111182a214d47 |
|
BLAKE2b-256 | 8e7b2b79d53aaece6e78a24d0e644aaa1c3729d756209de2990ace7ab6dd210f |
Hashes for blosc2-2.3.0-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8110d2f12337fd68c9a00ab3708ae3b394224e131c39461d8ef76d11a85f310 |
|
MD5 | b948a1873b56bfb24627a41e28e5486c |
|
BLAKE2b-256 | fe038766794a2c779047e863d475df01fddf7803affa2346a3c8b1d3eb685bc1 |
Hashes for blosc2-2.3.0-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8f161690474ed16afd7e71d5b35db051dce1c1e0bdc61f8e78f6084e0a68e99 |
|
MD5 | 91b101c5490948fbc1e67d7e58300472 |
|
BLAKE2b-256 | b9633d1f49fcfea9d2700c61fde53de3cb8a1097ab37b63ab8ecefc808a8825a |
Hashes for blosc2-2.3.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e6d9ec4bad030a08ed366fde7448456c949e3f4287686ffd8f394c11994f5dbf |
|
MD5 | 62041d348a28c5a8aebe142f484393f8 |
|
BLAKE2b-256 | 71aadc5fbb3dc67be83b9df2496d9a7c0cb48aca7cfb1fb4e93a123bbdc8f460 |
Hashes for blosc2-2.3.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37caf93bd6580b0bd80ceb22ce15b81fe085e1238a0dab57040ba371cfa45602 |
|
MD5 | da03741b49db994f645de7ccbe3e4732 |
|
BLAKE2b-256 | 246ab2588c734b2b78e2319e44a40fdd4e640385ec98b3da25d3dc7009c862b4 |
Hashes for blosc2-2.3.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb5c16a888757689a967deeddbca67bfc587e23efaf2cf12c38ee530ba071452 |
|
MD5 | 339ec4bea0811f23a253fe880456d574 |
|
BLAKE2b-256 | 960b7ab3a5e8142f4e0cc632d4df6393276c68a06a849c27d75324ed748f3b34 |
Hashes for blosc2-2.3.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fb9b1fb512be4a82fc17dc47d8d6fe5c8ab49abd609e29bd6a2fd7b151381e4 |
|
MD5 | 46889dc32214ea6cb1c2e37b21c7d50e |
|
BLAKE2b-256 | ce2c36f009dbacb2a9162a12fb104b360aa6e491fe988838ad2408b21a837e8e |
Hashes for blosc2-2.3.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfedb48c5ebdf22f40e2ed2332cc4bd0b13cf2ae9e9254070d9249065dff7040 |
|
MD5 | ace52116f61c494ca956054880a424af |
|
BLAKE2b-256 | 73fd1e8352b7778056ce454160fdba9eda462e30ca47ee8e1e37ce36585c3f2c |
Hashes for blosc2-2.3.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfc5f8c0dc040445c2205da63dcc8c656aac671e04dcee02b566434c5a175ac6 |
|
MD5 | 4fd7b427d79de8a4145222ae938cb416 |
|
BLAKE2b-256 | ac9419538804e191b227f28ab71a76fa80e16b9cccc39c8c8ba38d1ae338b795 |
Hashes for blosc2-2.3.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2dccfa9932e4ab2037122110e64aa9f3b213044821d2b1cc58477c7585c8077c |
|
MD5 | 3651f18c516997efdace46599bdb6fff |
|
BLAKE2b-256 | 390864ac3227c3e912ab3d295429eb5c072a2d765c65fde0cd29ee1ab6072248 |
Hashes for blosc2-2.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb37e557f7c670893aae304082ab021eb3a9d528efa775901cf340a9fdb7068a |
|
MD5 | 8c1994e264bfe0c9118abb21d33f6279 |
|
BLAKE2b-256 | 6f1db03ede7b5eb3a03fdb491bff9c262df36faf6f515bc288f2e13dec429c3f |
Hashes for blosc2-2.3.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a98155b3f2880c33e6ffd5aca9ec08d00f9b3e1451a0e31e031f1c1c04cf0ed4 |
|
MD5 | 8477d643b57dbc907690da182ada11a9 |
|
BLAKE2b-256 | 64ce93c8f2c0d206ab54c671916b300418965edf09eada773fbd20a26eb96212 |
Hashes for blosc2-2.3.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55a836e5a7055929bb0a9bf7a2794a3a87282d6e222327a32fe1f85d1d1fe322 |
|
MD5 | f8520132ba64b1e5f8fbb1cb81f71667 |
|
BLAKE2b-256 | 0963006d68ab59252f1bc7c088e4356c54bebe7611677947ca462303605f943b |
Hashes for blosc2-2.3.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2841b866615f4e92b4051b17c5d28123ce0501f7c5f34ddbdda452dccf19c405 |
|
MD5 | cd8c1265e3af605c8511599f78267186 |
|
BLAKE2b-256 | 240c9525b45018c351005a6a5512b99631a4fb5f0fbecdd7f60818be50664bfa |
Hashes for blosc2-2.3.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 430a9b175c88598844271d67bac92a077b59b6c96955170b3a7b16f634b4bd56 |
|
MD5 | 3571c9b2e70d3f7d40bbbf988b5b41e9 |
|
BLAKE2b-256 | 84b15fcf0a265a02bf8c928fa09bab4875cb1bbbd9acae05bfd8b9424a21de7b |
Hashes for blosc2-2.3.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de4ddce996c6d73f7bf6c172c5e6896a11708c6a0ce3e61dbaf7ba6ba4d055b7 |
|
MD5 | 19ff7eba27b23c89594a66da3eddb7ec |
|
BLAKE2b-256 | 41ff7ce9ba3425d5e4cd265b3b838d4efbfcbd6074bacebcc1dc604a07be2f53 |
Hashes for blosc2-2.3.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d10c41d5f7a7c8bb1ce06bef91b4f2728bd6488c188cfeab83c5e0815a8f6ff2 |
|
MD5 | 39df07ef8fad740898cc8d2e9ea35758 |
|
BLAKE2b-256 | db5d2c46e756ac8486f790253ede33895801bbc455c376df3bb87fb7efd47bb9 |
Hashes for blosc2-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e489c5356974f84b943c78f3591973069f003ea92a2cdf77cf4a20518b7e36f5 |
|
MD5 | a16fef480e9ccb7864d9f7761b4cb9f3 |
|
BLAKE2b-256 | 1b39500beb13f51f29d9dcb23fa17adb6e3bff682868eadd25bf4a48405942b0 |
Hashes for blosc2-2.3.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d6a5b12d005d6597e85d0d0b953eb441d660045fb56e4188a24ed9729330709 |
|
MD5 | ca07ab6fd67667cbe03f2501eb6ce51f |
|
BLAKE2b-256 | c0604944e43e60020a54803dd9be1ef0b5ec5ff941c12cd5c652490e90f90179 |
Hashes for blosc2-2.3.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e379249f7f50e9544cbc1c312f1317402ced90f7c6f94b00c023fc55fea83409 |
|
MD5 | 2ff3981139bde7b7d5a7ae84be8a7f71 |
|
BLAKE2b-256 | 4042c7088a9108d4d24b6e01545232a572e144123a010e460ed3123a469ab9fc |
Hashes for blosc2-2.3.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39cfda02e9624e04d695880b3f1d02cd70eca8ba5847443c57fb857e2dfebd65 |
|
MD5 | 90d27d69d9e2c6451c9fc3929e2d8960 |
|
BLAKE2b-256 | 007d957b839017c1198f9fb01acdab1d14e7f8199277a4fc2f7d92f7a7cf8d2c |
Hashes for blosc2-2.3.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97d32972a97ada1308ccabe788569f7c7f3470afce781720b5e4077550a42d01 |
|
MD5 | 4a14308d03cadec7874282e339606088 |
|
BLAKE2b-256 | 415c9053fc053d8edb358cc7c753d443fb7f49b5f64c0dd78f5cf9f8cac74cb9 |
Hashes for blosc2-2.3.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7c191a7ff1d365ef9606f126c639dbe4052793535b1217329d4d7e8c69ace4e |
|
MD5 | 30d59837dfdd70d57347f628cc0b232e |
|
BLAKE2b-256 | 4353a4885f443666d952cd5ad05fd66215beeda8d787b9767b921e03e2d1f885 |
Hashes for blosc2-2.3.0-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15d7b1c630b983a6baa682045d7d813e89b1a05e0c83d9b541a84a8a3d060b30 |
|
MD5 | 68cda38bb88bf05511a6160e1797c50b |
|
BLAKE2b-256 | d7ac46b6060039d150560a5d87b0a4584ac64a90610c5b307d2c29e12bcb642e |
Hashes for blosc2-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f69e2aa30dbfa9ae830c62bc0dcdc0fa0240005f03271d2daf38bc6bfc45150 |
|
MD5 | 4001235e226ccd0ae6454ba5b599e823 |
|
BLAKE2b-256 | 59f74ef0af7cfdd54a3b4aa2a813af440c1f4130bf66d1e2d40f788cbca3d873 |
Hashes for blosc2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d914071df65b62bd363f1f7833784447df4b39611eb972c9d770ce26ad0c8199 |
|
MD5 | e4934fb88c65fec7e3c13e0cffe77253 |
|
BLAKE2b-256 | 2c15a01d5bedbf86cacad7f81809b85ea9bb42c01cc2e826f639e865fae15e8c |
Hashes for blosc2-2.3.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa7fcbffb7b3d462cb91328464457f76888c1e68bcd1a256837419e2e8a1eb44 |
|
MD5 | 9b49146e3a2bbed84df786588944c52b |
|
BLAKE2b-256 | 2f447a1bd07fd090c677c7054e3ba8a632f35c030b61f050b3578690b6ed1a1b |
Hashes for blosc2-2.3.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48c4271651ec58c2e72fab13e4a59f7ac795dec833501d061eb3a340dbfb725c |
|
MD5 | 3f04ecdda082f36b625253a45ad22f5c |
|
BLAKE2b-256 | 0893c8b67a925582122fab1e3b9f0a6d55878377c0e999de771678f727e18e1e |