Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Enjoy data!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.2-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3ad209bf035b330184b65168c6f594fb2bf01cadfd0f0ffc0d6c680d960bcf4 |
|
MD5 | d787d54748194b21171a676e884ae454 |
|
BLAKE2b-256 | d4be494e0785fe51d4609b2fbb9922811f3d2957b04c12a35818428e67615f32 |
Hashes for blosc2-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 11c342e402e05f5c85856b98dd3e761dad2ea226139871e5e61ef8cf3dede434 |
|
MD5 | ca9fe63212bd1d25963a20aaf8a5648e |
|
BLAKE2b-256 | 7d08776f9b589b262a1f8bd13eb4fc37a72bcb59c0a982b0f362f59302ec4398 |
Hashes for blosc2-2.2.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31ddfc4ce288f54dddabc7494deb4005db33d0c702238e01b996403fd0844ec8 |
|
MD5 | 8cfb562954507b733e31983be4953ea4 |
|
BLAKE2b-256 | 8b56b86951d514c6506f79d7abdf0bc7fbd8defb18377dedf2f21ab86fdfd6bd |
Hashes for blosc2-2.2.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 031821a93b4b0c946a555d0cbc666b38d0c4b8f5620ae490a370253b868f7299 |
|
MD5 | 1433f982556b104221b62ad7d46b058c |
|
BLAKE2b-256 | 3d385fcecfba1655e078f5783674c5f8f8766e07dc3a2d252162e3f61fb9651f |
Hashes for blosc2-2.2.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7923315dda478a2e163b89b8be521211276fbed066918995c3b4288776473ef3 |
|
MD5 | db599103b302e0012dcad06e250846fb |
|
BLAKE2b-256 | ae2578ae5a87a126f6ace15649e7f9a70ce75552a7af32ffb604957775ec1dfb |
Hashes for blosc2-2.2.2-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10a98909ca1fb0b3b27c1a1b8f7bf6692383e72a21b7bf07ffbb843d7fb76671 |
|
MD5 | c51b6d2916968a62c8f6d3d1698dd336 |
|
BLAKE2b-256 | a9cf1d36294c87ebdfe1a2f7d6de2cde55fe0684f89e2874e11e7f65bf6208aa |
Hashes for blosc2-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d44ff6cc7613286326682fbda95f1383cb35d348247d57b1ca542e172fde842c |
|
MD5 | 7ffd05fe805d57622da222b62a95c655 |
|
BLAKE2b-256 | b1758f7e0508be21a3bcfda3bf6386b2a15a3a125eff3cbc4652fdd8e47af242 |
Hashes for blosc2-2.2.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c10da5a03e7bc5a5c4d0bdc6d3d7f7b738d79d99f58e3e02dab40c7d1f46a000 |
|
MD5 | 429e6d0d757dce2c2822f815b3d860ca |
|
BLAKE2b-256 | 3a901043797001acb21274e65519cd3924f6d115ecfbc9b011f3f88f55a58d4c |
Hashes for blosc2-2.2.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a096d562036ca4925a815be1c889af2d7dc5d1473a435ac7ffe14a22a31bbaf7 |
|
MD5 | aa62e8274560643db9899012d2f0e84c |
|
BLAKE2b-256 | 1abb405b28047c895a9490140250b99f3592fa1db460fb2e1bb3ae2867d2906f |
Hashes for blosc2-2.2.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b9179bcac964664fdec49fb5a19082a8d470fad9a9c5f34c8ae0b35b232abfb |
|
MD5 | 3d2603fc99589f1d996ede3cd87825fd |
|
BLAKE2b-256 | 11288d05616c385ffc4a51121799841fd4aca354328946b57eeea3822a138c1e |
Hashes for blosc2-2.2.2-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf4996834d4730b8e7bc9f944910e76a30d8cd715348fa81389297b2a5ddf403 |
|
MD5 | bf74cb9f296076d4121ac6601aef8fe3 |
|
BLAKE2b-256 | 063fd69cbd37e8b378afe6bfb2acde80ad0f3e8bbc70d53fe8b6225d006f898d |
Hashes for blosc2-2.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fb575e37aee0a6bb9b2f775b0ef7bb03e18038baea3aec143fdde0f7766b16a |
|
MD5 | aa8c6b3bfaf88ab644b220c5b57ab112 |
|
BLAKE2b-256 | 1ff2a68e898699675c64bf0323768456c7058cc83ba310d766cf410cf1f016df |
Hashes for blosc2-2.2.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da2b29ab3ecda3094d516c5746bf3cf19759aa353f571955620a1ed5c9aa48a0 |
|
MD5 | 930f197436a75166da2baeae1f06fdeb |
|
BLAKE2b-256 | 543e11f12c403c8d3571c33e41f161e0cd64579ae744f36056b073bf11fca90c |
Hashes for blosc2-2.2.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10513e34e91020ea9eb7677f041fbbb79fba341340a05d232a05242bfbfca153 |
|
MD5 | 8d1b672c74fb01a0890933b063396078 |
|
BLAKE2b-256 | c76f030a1616d16d57ff35008779644fabb545fb4913e64bad66d7d5fde74d3f |
Hashes for blosc2-2.2.2-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d392c52dc27720ebb7bb6f54a2c35f2474c9969c6801cadff05b9c9a30103406 |
|
MD5 | 92c9e50fd60072dced76b641ffd372d5 |
|
BLAKE2b-256 | fb417847a2b5b3184af1270003b15062a40d691d879985caa51bae2d5e51b838 |
Hashes for blosc2-2.2.2-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fde2834fb4aafe1a3fce75a5f97604887b2607a3579eae9bea5acfb04abb0b0 |
|
MD5 | 1fe02b49d12fd05e09da7ec9b4f5a00f |
|
BLAKE2b-256 | bd5a30244a9bfd42520df2e8a04561bd492ebe2c873a21813cd43f13dca807de |
Hashes for blosc2-2.2.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6356aceae0e560f43f650441b68b89698ace61957d9f1a6644475f9503735261 |
|
MD5 | 21a1a65773922a3f2bc93812010a0bff |
|
BLAKE2b-256 | 570e3e066e802f6845a3a006af0529a0828e44e31d2cd4b97b1ffe8fafb77c48 |
Hashes for blosc2-2.2.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 113b68c7bb37c1ad458de8a0b3c6fac933b4f505bbb693d3854aaa43f03a4ac8 |
|
MD5 | a83351540ab1f150549b9d6613f98789 |
|
BLAKE2b-256 | cd95ec9df6ef4fde4dffc1fc0ffdd8faca21bf5dec40a36d5b9bdc59b41d7567 |
Hashes for blosc2-2.2.2-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e605bd2a33e1e206a987ebf7328c09d95baf378f8cc0b19c771779328e2f3c58 |
|
MD5 | a80f23536280abfede704f3b6a32a09d |
|
BLAKE2b-256 | 3bfbeb7d355b960b1cf0282f9cb5851cb4b1d4dedd2b369854c5337ed6526b2a |
Hashes for blosc2-2.2.2-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f881f6d053ac46d387e7c44adf68e7337217c7e0de0a987bc338333f9dc08abc |
|
MD5 | 64a48ba2b89ffa1622894874fcae326d |
|
BLAKE2b-256 | 1b3d6fd41844b58aebcca5366de54c4e408cfa8c98d1e66c118c4dd49969969a |