Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.5.1-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 585d780c5e85f251dec72b75a47666e4a261dbfe1d228769bca545e9fe07f480 |
|
MD5 | 1f888aec64759f779a1a06e6ee2c040d |
|
BLAKE2b-256 | 2980871cf959e5e0d3d2a177caf7b9f4b714850a50c33bb5cedd86a95c6e05b2 |
Hashes for blosc2-2.5.1-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5856e57e0e81f9018f1a12e803b9f768fa5533175092d72d165ac60069c7d2ab |
|
MD5 | 571c2cb995f321d45740d1a87bfaf042 |
|
BLAKE2b-256 | a1e6e05e987db27986ca8bdaaf442b94212e116a1eb937a8cd35250548704aa6 |
Hashes for blosc2-2.5.1-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 956a63231f1b448803e9b4bc3e704ea424c89fc14418d99093472c74f19c19e1 |
|
MD5 | 4c114faf5179d3b272e9f5c98fda8c95 |
|
BLAKE2b-256 | 7821f1815f711f98c04eba5116d55041b2f64e17e8a9437d7c832d1168ed2fa8 |
Hashes for blosc2-2.5.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19f79071a336fcf1eda01cd0171291a4ab82b16cf9a15d2b4d26c010146f13b5 |
|
MD5 | eced81e9062b27f357f1977cdd0ec7d0 |
|
BLAKE2b-256 | 33bf52c8385aa71ed8c42296016d48c80dcc41dd005e951a83970bde4e6d4ff2 |
Hashes for blosc2-2.5.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb5fcd1775b3884d9825aa51fb45253f45cfa21c77f4135fad5dc5db710c2a34 |
|
MD5 | 3365c81b2f9ef15bda505dcb386fbff7 |
|
BLAKE2b-256 | 928db13ea33ea4d5da344d1170638b2d7b3cc63a921b97f9a8184128fd78aacd |
Hashes for blosc2-2.5.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd66e60dafcc93d4c1f815d726d76f9fb067ecc9106a6c661010e709135c79ce |
|
MD5 | a14b484179e9f64ba8631d0c110b9c1a |
|
BLAKE2b-256 | 9b219a887c9fdc46a5cfe4aa250eac4ed4f18213bf1996e113c1d1c662e31678 |
Hashes for blosc2-2.5.1-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab849d3adaeb035f2f16cf495cff1792b28d58dfb3de21b9459ee355c6bb8df3 |
|
MD5 | 81b99bb592b53a48a80372c427e9d2b0 |
|
BLAKE2b-256 | 5b4d00824754a3b5b5c6ffb92a043b5122be6202039fc76a25fa913fe0d4e235 |
Hashes for blosc2-2.5.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7499e277c13334d54f84e74f429f32341f99f7b978deaf9a7c2e963904cb48c |
|
MD5 | 58f507a4c824de779660d7cef83d0b17 |
|
BLAKE2b-256 | d1487d4be4e57342ec60c05d11478257b235cf33dc96c31049fb47e6ae98b652 |
Hashes for blosc2-2.5.1-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a3edc3256bad04d3db30c9de7eac3a820f96e741fc754cdabb6a9991e5c37e8 |
|
MD5 | daa75bfa97fe6d712727077f3a7ac40b |
|
BLAKE2b-256 | 9f3921d6d1dc00d765f5c1671f8c0b323096f0dde360e8e1d8005160679b4831 |
Hashes for blosc2-2.5.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0fd109eef815ea1e50fde4f676388aa2f3bb5543502d125fb63f16ec7a014464 |
|
MD5 | fb96143b49fb8840abe4b26497292c6e |
|
BLAKE2b-256 | 8f9b923e29af25f4c34b5b331829a87bc8397b18e8a361551230b70e6b0a785a |
Hashes for blosc2-2.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22f03a723130cf07e4309fe34b1360c868f4376e862f8ff664eb40d019fdd3f6 |
|
MD5 | c33b40dce0158ed2dc040a6c01b68bca |
|
BLAKE2b-256 | 2a0035169a0765044be69a3bfb73441ff8b9ea05d92d4660be622bbf90c46913 |
Hashes for blosc2-2.5.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af3cab9c12a4364c643266ee7d9583b526c0f484a291d72ec6efb09ea7ffbbf9 |
|
MD5 | a01ec9d195bb38b3e9f034b363bc7336 |
|
BLAKE2b-256 | 1726d34d521fc60b164738d986144642c58675bcdc344cc9fbb8c08e58bb11ab |
Hashes for blosc2-2.5.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae2e0c5dc8561a6b17842ee4320b49621434c20e622c9e9f5c67c9c6eb3b06a3 |
|
MD5 | 894ac4255e9b941ad013244e25ab5ff6 |
|
BLAKE2b-256 | 0350b1d2bad6b9f77c3540b511b5dee77f213c8b179e942bbc34f5c410b92ede |
Hashes for blosc2-2.5.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da826d42d616f8a939f27e1501b40e764fded66bc80177eeaefcebdbf3b3afb8 |
|
MD5 | 33bc41937e618d123107cda20f54d0ef |
|
BLAKE2b-256 | c3f425a82b7191109beb0fc20c580f35667a2e86087ebac2f6a9e50469f618d7 |
Hashes for blosc2-2.5.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5a7ef00b82fbca069e949335f9c92ce7cbe2039a9fa2e2bd4f5f418043d6262 |
|
MD5 | bbd012bd19c13eecb4a989bcd9ccad01 |
|
BLAKE2b-256 | 67ff0cf994e434b5ae860546cfe6941c58bdcdc27ed10009caffac9a29b66651 |
Hashes for blosc2-2.5.1-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 873483bd5c6afb8d139039180ee57b74373232e87b032cb80389fd8bb883ea8e |
|
MD5 | 5c176ccd990d328be39f324c74412241 |
|
BLAKE2b-256 | 9bcd56150f9367516f039b62923a5cb243771e2a1f440de0232b195f6e79f1ab |
Hashes for blosc2-2.5.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4dc4f595bf95c350c50bb77a8749cdd08a5dc2bdf3bdb18983d49a52d60b595 |
|
MD5 | db6ce853144c0c5cfae41f01ea7c3cde |
|
BLAKE2b-256 | 167dc81b1caa0aded9f3e63c4dbab7a334681ded12ad80a18de34e7199939df5 |
Hashes for blosc2-2.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5455af77e7e94159bb4966cae554f232ca2d52bb80cd3f878ecef39cf569da2a |
|
MD5 | 1a798ab4402b49eccb0a7a9afb3ef75a |
|
BLAKE2b-256 | a58cd4ab68a40004f93c1fd20dceb1899e54477425542bffa1f5ebab2647956f |
Hashes for blosc2-2.5.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 546fa39f397dd54b13d7c42a4f890afaf16c70fe478712070942d464c440ce03 |
|
MD5 | 86e8086705533b0ffc72e761e2b9d8ee |
|
BLAKE2b-256 | 327232bbaf583eea51b27c91ddb145de017d970ec0a835ec70e966925b00c9cf |
Hashes for blosc2-2.5.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f35b5d69a7a41e9d5054297d2540c25f8af5ea3c62e4a80ca7359292d783c04 |
|
MD5 | 80752c18855b69c6b554e578fc97e120 |
|
BLAKE2b-256 | c08ac7ce322f90750eb7fdde0948e5c6b9c47ed03e276f2481b701f9d2360d3a |
Hashes for blosc2-2.5.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c861262b7fe317c1614a9b59b6c9edf409532b4a6aaf5b2f4ad0d79c6f800b57 |
|
MD5 | 96fc679a0c1baf2a5825dd1c1c0007c5 |
|
BLAKE2b-256 | b2754511f1f9cea0aad8c5464736720a9de1e762e60e359cde6b7d69186e97b0 |
Hashes for blosc2-2.5.1-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73eb5e569a91fbe67f7dd78efe6a1ca9a54afff2c847db5dfa675bfd6a424f60 |
|
MD5 | a02aa95cb1f2dfe87fbcff2f6a880cfa |
|
BLAKE2b-256 | 5cc1f80115c66a181e2ff18027818d3e1f37a3d5133b350b3e25189e53e28cd8 |
Hashes for blosc2-2.5.1-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4315ae8d467fe91efa0dbe22004e967008f5fe021ebb3945518f5213d7c4511f |
|
MD5 | 7031dbda066e3bf7c755a3510cd2218f |
|
BLAKE2b-256 | b57c6a7d46a6094a100e1bdfba7e1df584971efea00732006fdddcf0320d4846 |
Hashes for blosc2-2.5.1-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7afe59d35d93bf8da7db8de43f4d8aef277514de43953c1e5e416ca839b9023a |
|
MD5 | 881ad4963b032a1055e3d755b98d9519 |
|
BLAKE2b-256 | 6288a185ab2a50b930011396b601ae8e82e83dfa24597e5f1fd91f483f178a94 |
Hashes for blosc2-2.5.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b473472b977b770aab3bf20d0feeee84ecd5bb8b15a675287e090ce818c1cd40 |
|
MD5 | 472b7b4d1b15d5a787a39373a917858a |
|
BLAKE2b-256 | a6e2a1367ad174006e5625a1c8ef1b526fc135adc6f546bfc46264ac8e508ebe |
Hashes for blosc2-2.5.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf6efecc1a22da26c73ff5c60d0dc086db1e7edcceb6b360dd193cda893bef28 |
|
MD5 | fda4469815b2b35f698c16cd438f863c |
|
BLAKE2b-256 | 65c40c9740c4f5efb7adde87ed8b5af88c171a3ec007df912cc2fbfb2736963d |
Hashes for blosc2-2.5.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3def4650faa1db43143d821228ef58797108cc95d6698c4b1581909cc2b149ca |
|
MD5 | aeef51196dc71e0954ffcf57758253ac |
|
BLAKE2b-256 | 07cbe97de5b2c40b6c2106d5b68c1d823707a16386fc08781a6c7c609000866e |
Hashes for blosc2-2.5.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0cb9a6ac1abc466c12bdc90052f17545512de8f854e672a1ea4d2b40292323f5 |
|
MD5 | 7165609e14e637e164d587822b125d43 |
|
BLAKE2b-256 | c94b799dcec670a1d511d11a40e919241c26a273cfd2305a4f3fe7444c39758a |