Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.7.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62d2a6eaf1be1858993a4d7b2b8efd2ede5c4eaabe030c611cd075d907aa5400 |
|
MD5 | 6ceeb562c0779e9aa5b6ec9b1622add1 |
|
BLAKE2b-256 | 60bf6ceba93b39a716bca4feabccb034292f69fc06fb02bb86ecdb828cb95881 |
Hashes for blosc2-2.7.0-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 407896867032a760dcce6c25d5e5a56b6fe5235245e065e2549697f69b5117c6 |
|
MD5 | 512d1181e8051402746dc0917c309fa6 |
|
BLAKE2b-256 | 3e6be23649ecfe45d58558f11d67abff6ce390ecdf612f5b1f698a49d44127ab |
Hashes for blosc2-2.7.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1546c04d25ce793fa0fd7a83999bbb576ff84ef474fb45801f0b6dd76b84803c |
|
MD5 | aedf4bf6f582cd43cf67b4d9bbd24ed0 |
|
BLAKE2b-256 | b945935e902158ab55df8d4c764ded202a9f32dfdbe683e827cb17ec2379a78c |
Hashes for blosc2-2.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6a700f9324b37e814c5633c43b081c60962f4dd59c0340cefe5f61f9f0411fd |
|
MD5 | 30de13d2ecad9dc9a54cb20c9744b588 |
|
BLAKE2b-256 | 95038e7aaf9295c490b495ef9663d4e6a53b7b9b669c856425275a8a2ed6dc10 |
Hashes for blosc2-2.7.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 770733ce68d82674d1f80961fe56f3c2d914d8ea4de036af3888a22479add97d |
|
MD5 | 89aad828349a29e26861d8ef3762bd1d |
|
BLAKE2b-256 | f94159e615cc6fd04a3c7a9c28cf082eadde713fb50c286f4c89335981b0e4fa |
Hashes for blosc2-2.7.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 64a26c9f7a4a5ddc5721a75b37f913f9e21c0dab96d8c152a64f8faf8659e9ee |
|
MD5 | 178f179ccb1d4e317bf021877feaf061 |
|
BLAKE2b-256 | f65f3bfc80bf16302a290324bd3e44512702a38e60a9d0d1cc26de0f18e46fe2 |
Hashes for blosc2-2.7.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 729305b06e76b0c95b0ea5090aa7ec87eff72ca43e194283e0cccee92bbdd1e6 |
|
MD5 | d05bcfd40e77e806453ec99e62534b97 |
|
BLAKE2b-256 | 09d1a17c42d6d2590bdbe456b77b3d087e7263c218c1737273bb47cc3eddbbe5 |
Hashes for blosc2-2.7.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05d40ede9cf0ecb25500cfe9bebe190e75f246eb1fcd7bd358ac1acfef44ee7a |
|
MD5 | bf41b6858b0bb309ac2648b78a88d0e4 |
|
BLAKE2b-256 | 0d4ab013e712956794c8d175b10dd89a932d11b6901bbd26a290577193fef2e2 |
Hashes for blosc2-2.7.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4518944374880d822f9ca90d4473bfa9f4d884b462f78365e224c2b291962e44 |
|
MD5 | 2adab866db725687ec96d7308eefad56 |
|
BLAKE2b-256 | 330c204d65873250e4409fd3e5611e54d115e301fd7ea2929d72006d4a290b2f |
Hashes for blosc2-2.7.0-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd3ca9a61bce4e4dc8006b613fa9dd8982f71e01fa9f593d6cc44d9fdbb56174 |
|
MD5 | 488b7f6518f9529b2073e6661e748823 |
|
BLAKE2b-256 | a94a8b9db123d7a514eccd9c366aa9f14103b726226ee4e1eca33a4b8d8af459 |
Hashes for blosc2-2.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d4b208d5f5947d3062d3353717c43e0ea8e6ccdecdcd30737d5305628e0062b |
|
MD5 | 0022ccb4804cf1c696c9093f241af266 |
|
BLAKE2b-256 | 365de74303d76a81b593084dbb6b2767375ecc1e007d7e877f4bf4552662d8aa |
Hashes for blosc2-2.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 634bc22f17ae47a166b8201c77ba11bc160d9997ace51fc820cb3cbd285d47f8 |
|
MD5 | 5cb5d2de86774c19fb1748305287f0fd |
|
BLAKE2b-256 | b8cd1a8aab659f7688c277a2f3655414d7a5564ce701bcbb0db4dfdcf09e543c |
Hashes for blosc2-2.7.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b640fe2d1d39af2dccffe5e100ef94d21940bfb7f0af44ba17fef718671b267 |
|
MD5 | 5862616d5a459f5f7c1b060aa5a76878 |
|
BLAKE2b-256 | 8a2418d1925f9518db6beb77b26073712b208a318931e9bfe7c142c54afdd1f2 |
Hashes for blosc2-2.7.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 565701ad336946a7ef12250def97aae2257de1da34ac8cd570be91b664a03d30 |
|
MD5 | 40da42487a7f159e1e45b32d17340b96 |
|
BLAKE2b-256 | eea727ae33a06b4d83838a6675b433f3fed4cbe13dea6488bafa6af235a435b8 |
Hashes for blosc2-2.7.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17dd39f62f1686a170232ac8bcba40358ef67e919a91fe840ac71a45d067df30 |
|
MD5 | b51e66821fd6737d35df125e2cafc9fa |
|
BLAKE2b-256 | e4f13e637d84af04ba6feb81c0cdceeafa91f3493a886c28260f3d9ae8dc617f |
Hashes for blosc2-2.7.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 488dc4be3b6894967a7189952634644f8da46c4bab7734719d379cdf5b440dc0 |
|
MD5 | 1b752f0d724d3cb748c5c77e34366806 |
|
BLAKE2b-256 | 8dcc7a72841f5ed357f6dc0af208879658bf22d58148d05f2bf0e67f65773bb9 |
Hashes for blosc2-2.7.0-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9518b7bbaa0f9903a5a921abe6abb0faa56b0e0ad2da0416ff3a486a4b2e0aa |
|
MD5 | 6f5a89703c042c662893d2769c24543d |
|
BLAKE2b-256 | 1be21ce65e3c7295c1fe3edd98caff58f4aff27d0d6f9ba06df233d2df13c216 |
Hashes for blosc2-2.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 928a89851b8528ce9c233048d832be5b2fef47645d5a389c021f3f58333fa3f8 |
|
MD5 | a08d64ef33665fd5363b44ab8a7083d4 |
|
BLAKE2b-256 | 44708df5f8a79c7b9019654e3d890d4ffcab740087ccb88c5df05f7f79ff98b6 |
Hashes for blosc2-2.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1ac66ce25214b0b2e53beda9bc6f333dba16f2667649b1026ae041511b5a07d |
|
MD5 | 22b7165ff1e99a7f78ba746df2e4cfa4 |
|
BLAKE2b-256 | cb250adeedb1ae43f40d4e4d1b093dc1606a79b5abe4a49ca671390aa1c7b06d |
Hashes for blosc2-2.7.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18e3c4c95fe40ea9cda88c784d96e4efc8ddf53f94074cf46daa2e91c9ae5137 |
|
MD5 | 8827596054b33db1d02d76422eb79a67 |
|
BLAKE2b-256 | d534c14ff112a6a7ac6586136e6dc07ba0b9e29b144b9a9cbf7b5db48f894364 |
Hashes for blosc2-2.7.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa71042277956199676169335eb64aa76e33adac5a22289eccdb7d10edf402b6 |
|
MD5 | 2b6fa87d84a447a8d69e2ca364954495 |
|
BLAKE2b-256 | b1555577554cda74e7c0508035e3b15a8cf514d731548f67c526d18d1e2256b2 |