Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.9-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0eb8ae893b60743a31feb4ed02dd96039400fb8e7fc5ff4d9adea8d70acde204 |
|
MD5 | ce6ca15dd7c403bf10a976689adf0219 |
|
BLAKE2b-256 | fc3594cb65def25b2227e9ac2c3cc9018664c705ad573a9074a222996ba22e30 |
Hashes for blosc2-2.2.9-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 658443f639975d29eaa3feea269a2f971d2da5cab736bb6462561d7efe261cc3 |
|
MD5 | d0b09e89e39f0f4268a0172d88bf2203 |
|
BLAKE2b-256 | 435e095d2822fd9627a9cdb8bacc8772105844a89a93f0c5271cd269797bac7a |
Hashes for blosc2-2.2.9-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a46f9216d63958572514354b94eaedaa2052b60b3301ec7c41c8f30c6825c718 |
|
MD5 | 5b7d39543c411a67ccd3cf2f9c002d0f |
|
BLAKE2b-256 | 99ecc392aa6a103ca8746fbe2fa2998a8c1fb4a4c73edc3b180eaea6f9f34949 |
Hashes for blosc2-2.2.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | adaef04627713e22bc7883a35afd499266762f700d8644a65cfafbf2879d4350 |
|
MD5 | 09379ea9dd8cd98a4ea4668e692254e6 |
|
BLAKE2b-256 | 6b8739f54f45715b97a466407f587238c186006ceb7fa84e500b29d079fe80bd |
Hashes for blosc2-2.2.9-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f10e14c7f3b9f14431df58f9891e490af83ae6fb3d7c2a7d05722560273a2da8 |
|
MD5 | 1876fdaebacfa47fd53fd7671acb339e |
|
BLAKE2b-256 | 1704498cadd950eba17ace95c56f3578a4341e15d2a3174af41e24936d2124ad |
Hashes for blosc2-2.2.9-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e38cc441798595f05e70d620f1124cd4c472003f9b58c17e79dd0477a4d151fb |
|
MD5 | 108c724177605a5a515576e7173234e9 |
|
BLAKE2b-256 | 59a06d7cdc6b0db4a54b68fc890e1724e79e1bd25e20ea33ba15d0fcf07518ba |
Hashes for blosc2-2.2.9-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8504a92404b2ba5112db83bebdfbe7eb3c286514acb658191434f020ea084c7a |
|
MD5 | 91201ed2e85dd4ab1fc60336c8190f96 |
|
BLAKE2b-256 | 078bc75991688dd128238904901320a995ba68323e6dc5d98f0b0e25c28818a3 |
Hashes for blosc2-2.2.9-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 368b12e43249e55137a05506e747cc4656539afc73bf82a85b896a2f13a529d8 |
|
MD5 | 12eb83bf58a69ae68396de5756d37553 |
|
BLAKE2b-256 | b84b880e70fa21d1f38db1204f0bcd573debb514c5b4d264d0138f5f04d3bc1e |
Hashes for blosc2-2.2.9-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebfc1e9736d83bffa16e49f53278de6caa7b5469c44a4448800fc40009efbbba |
|
MD5 | 04e59171f1fc0a3cec84c9304221082b |
|
BLAKE2b-256 | 560b616d435f42f2571581ced4e6e3a0cdff69ee01078060373d725fdf52bce2 |
Hashes for blosc2-2.2.9-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a4db24030be00e8ccc9ff0645716504e4caf7525b70c7976ad8434b47f04f4f |
|
MD5 | ec422c182f56af5c48b297be8bc78612 |
|
BLAKE2b-256 | 9abbabbdade142446acae1e518f202d774d9bd9efd06fc27b0a1e6da2c3042f1 |
Hashes for blosc2-2.2.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c11ace31c542aa6eed11708e7b92cf5d3dbbb3c1b8a691919c3bb6130caf1746 |
|
MD5 | a4b7c51683cdb9939e1e809d86b666f9 |
|
BLAKE2b-256 | 84add23ad6d09b624bf75865a13738fb2993e9a51f8d7f6f41c58b0d64995d37 |
Hashes for blosc2-2.2.9-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e82b6280107b9ec05aa0ae7d86a3f73d14bd99767901cec95dab622d37cb0d7e |
|
MD5 | d27791cf297bc85fd941ffe1b5b902da |
|
BLAKE2b-256 | 1f6960e0af47f852214338fd722e21b1fb2a015a54520a57b83e1aae855021da |
Hashes for blosc2-2.2.9-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7746244318adeb552cfb45c95b329eb12e146159ae6506b06b4854dec4c3b2c1 |
|
MD5 | d86a0e84eb2fe76bc2db43f381ff08df |
|
BLAKE2b-256 | f137faefff2e27eca407eb92580d78df2e941a0976abed4f40169c67519a77c4 |
Hashes for blosc2-2.2.9-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f774b0c20b86c99fe1ba4fa7737add60d71930662192fdf66a547707a1e3a37 |
|
MD5 | a85b8cd525402317fc183c436d0b380b |
|
BLAKE2b-256 | 85ba9c1f1c8a4dc686f70cf9721e4d9fdb1fc01e2fc2f4f517acfb3c8bcc6185 |
Hashes for blosc2-2.2.9-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e24335d97ae43558d222b15141d8499c3b220b3d166350441a6d2a4470997921 |
|
MD5 | bee9c8d5034afd12dc6740083ca6e449 |
|
BLAKE2b-256 | 5703f21cda952e6571c38b3af38acbb11ef41e685561919b5bdb69b237f63e33 |
Hashes for blosc2-2.2.9-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49f3b3951764ddf6d7ad3c1c0800adef2b7780348b1fe5126b6e0970f3ea6c2f |
|
MD5 | 9d0315ca989707c9dd8d0ca23443bc44 |
|
BLAKE2b-256 | fe5987a283ca28e8cf72e1babd52bc7ff0e20d5d6a18de0900c1739b718d82fa |
Hashes for blosc2-2.2.9-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73c7a7afd5390d60ad8ecd1e0e5de2492c60a24cce748b8ae2da83ceda0649ad |
|
MD5 | b4ded000c7db8011c76a1dbfa27b2c41 |
|
BLAKE2b-256 | 30636e8a7876f05ea3ae0bee57166d16cdeddf8e3916470271b008402d68ce10 |
Hashes for blosc2-2.2.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74a24b4efb8b608b71d8af51d5c8f16dc63f45c2145240e7d313472fa720a68e |
|
MD5 | 11b9c0960c2cdc6193fc4ea1672d79f2 |
|
BLAKE2b-256 | 14b839d5d06300785c2a0f141e95a27c959acf7e8e2d98ff93de80c818598181 |
Hashes for blosc2-2.2.9-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f9413d6926d7442847b115680567fd4ad4ddcdf46e2419cd2f5e82ee8d00f6c |
|
MD5 | 9a4f85b0839b2fe17c197eb126c2646a |
|
BLAKE2b-256 | 04713ed26e1df89d739edcd5e51fd8cdbfba0c4113a40eacac1b38dbe38824e0 |
Hashes for blosc2-2.2.9-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 562828192e3c6f4629823d836bec1d129dfdad38a7e6d2e84f52dcaf9979633b |
|
MD5 | 6fb24a3a844405485fcbad6f0d826e71 |
|
BLAKE2b-256 | 95b142097393b8ab9ad423f7119a30fcc4694822da1bf870c68f4cc32c127617 |
Hashes for blosc2-2.2.9-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 135afe34913cd43b02186fb400f30e2c9bdbfe3752470d9b6b00a20e7293fb9f |
|
MD5 | 593fc943999e300eeba3da034dddab31 |
|
BLAKE2b-256 | c864863c9f0c6414e919132a893d9f56dc49dbf6ca1b951a261d86196a77393a |
Hashes for blosc2-2.2.9-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c840bdfd97e25cd61d6e048f8d9ee6478133f3e70c880c2cb3054db93e142bba |
|
MD5 | 5a145889b580c7e69139f6b4320dcf3a |
|
BLAKE2b-256 | 35754ffe30ec8aca7490546440d05836a14fd0b56b3f53fcc9652e76595475ed |
Hashes for blosc2-2.2.9-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa36fa18b8d41aee7db975a318b481304e6e3558b48641ec53933287274a4ec3 |
|
MD5 | 63bbf8cf9f56f2ca0d2fc3d9f11b3243 |
|
BLAKE2b-256 | f16ade9704f555e6941e34c5316e913128bd5542955cdf22b75e5b6826ebe474 |
Hashes for blosc2-2.2.9-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25f27b50b2823e6a2e142eff02840979c19f629eb7833b45a98332a2d728543f |
|
MD5 | bfe1652f8b1380a8f67f75fcbecc6af8 |
|
BLAKE2b-256 | a5beb4cab7bd7415bdaf6bda7728674d6cf2dba10f06d4a73cf8ddcf018dbe61 |
Hashes for blosc2-2.2.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f31c0ee147f5f78ceeb65b601c47b0431a0f6111b8443aeb1485547394725895 |
|
MD5 | 529f8929bff74979b99b1b395400cd6d |
|
BLAKE2b-256 | 6013505b992b99b44662d8e710e6eba13b145c27effdf897e5adbbff374e3ed4 |
Hashes for blosc2-2.2.9-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97e788170a2e80cac38f15d723f7397a87d3c522980fc4f8d96c6fa9f5a74dd3 |
|
MD5 | 8a9d9b40b2f92bc27c8223ae26437160 |
|
BLAKE2b-256 | ad9b2267495bbca1b88ade2090e49342f0726858c50ab1a5a435fb956e500bc0 |
Hashes for blosc2-2.2.9-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 555468f4c77a45e35a7a878fab7679bf4705585a84b81649fc423eba293cf17b |
|
MD5 | 97396249ec7f892dffde8b9345bca9d4 |
|
BLAKE2b-256 | 7c50e3a7ee184c80c87d03122b524262a179b15752ce3c8fcb1f8c30cf32baab |
Hashes for blosc2-2.2.9-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82ec6d1a4343868ce833380c82f60e9799794e04d35f630af948f0f3d28c3577 |
|
MD5 | ed8842c4a80ad889be875eceb7ceaa1d |
|
BLAKE2b-256 | a2bb81d7eea999ae811756d76b23c79c7f1a1d311b4ed981924a65b5f92a8ab5 |