Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.4.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5fef27217fac665eed97e30b39c7ca1f810637ea16637ae155d80df4fb5017cd |
|
MD5 | aa9363b5a3937c0ce57121e833bfb398 |
|
BLAKE2b-256 | 75a0245d0fee26a9a0362977f425a886914a0cb326f403f2832beef0b2b45336 |
Hashes for blosc2-2.4.0-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1249a39b2d5ca7ec755f396108efb209d153130afe6783d04aeb2f9a828a4b67 |
|
MD5 | a6c7bca1601d1c79d01fc1c25956f928 |
|
BLAKE2b-256 | 92d16372e39dfb3fcc9c74753b8fc286585d99c02241442f27d8253439df74eb |
Hashes for blosc2-2.4.0-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c05c6748d71be867b5e66fb0915c97e3852e1cf9bcb4af8eb9cf1d2a6c517a1 |
|
MD5 | 575612d1ff6d1019c1b2c0515a9e0c96 |
|
BLAKE2b-256 | 9bdccf6099fb557938db07587143808440f9bad89f8b9af646b83ae7a3ec15cd |
Hashes for blosc2-2.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2292affdf9a0056e1e60e949c3de1a3c47603b2b5e3b7d9ebf96deca468aacdb |
|
MD5 | c864d65506e82005fd23658bcc65f11c |
|
BLAKE2b-256 | b090e08b28f37f66aefc03341bd1092bb02bcc6b9a1125ed32716f9f711060de |
Hashes for blosc2-2.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfd17a8b533d64320e636f67d68a4361febc6c18957d84022f0cc2cf6c0da77b |
|
MD5 | 99ca974c3bcecb8cf8b218c444e3ce37 |
|
BLAKE2b-256 | d3fea79225d1cde0ab9221ed0dd25bb4efd572d391b245aabd1b8303b0022cec |
Hashes for blosc2-2.4.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a180ce8c49b8897eac60e51f990c93783dc4a8f5e3dc0dcc29ed4f60ddfe291 |
|
MD5 | 7da016c0e7728e2b488c34e9e5e88b94 |
|
BLAKE2b-256 | cda3a4d1f1984b135aabb7454437e726ff87b2ecfa85771f07eb73a541fab591 |
Hashes for blosc2-2.4.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9cefd054da030d86b318c9b0d6383ba831e7ea0a11bad7bd413ae42e1ef755f |
|
MD5 | fa912b788e19fbdb7da36248048aba1d |
|
BLAKE2b-256 | bb55d2d9efb2990005aceab4df1648e3e440d4018d8870874a071243e30ce688 |
Hashes for blosc2-2.4.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09e4ada928d54ebcac77d66265db030d235466b2e5a43001859bd97730658528 |
|
MD5 | 6a2bad96438e065a8a515c7cded96863 |
|
BLAKE2b-256 | 9c2468e19daabf0601452ae236b31f54d21b3796180b7d7f1a9d4cb6d8f1a649 |
Hashes for blosc2-2.4.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0958f899238507485d2a260b6bb21e06e20e0841acfe73caf04697dc758d5cc3 |
|
MD5 | 35007d896dc130cf34d21840d29be4ef |
|
BLAKE2b-256 | de427d509e939e19ad7bced2f57b5746bfd98e5b76a84c9ecd7071a44fc19530 |
Hashes for blosc2-2.4.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3e4e64317255bf56b6349e817d6eaa4f25c520f40996d1dd604f1337e5b8e80 |
|
MD5 | 0b78682a3d1cfebf230446f298cd4438 |
|
BLAKE2b-256 | 2cc16fea931ca64e63bb36616a55b0f271a85213ec099cf2ef4c0c52c23fa107 |
Hashes for blosc2-2.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43170e534d9e2d2822dabcde0a993b978999f9c68080292b74eac9502257c7dd |
|
MD5 | ac7a4939f0beabe4b46946bbf1ca1cbe |
|
BLAKE2b-256 | 5389e0ab7a736cc3e191ba51d665e4f9e58152152817b6ea6a62a994929f1644 |
Hashes for blosc2-2.4.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5334e9a48f1cab2246fe08c535182b61f1786cd7d50164551f0115e6cdf45457 |
|
MD5 | 0d2c39b83a0691b505da6bf887f0f4c0 |
|
BLAKE2b-256 | 714585acc5558ef7684121838ba1173808f23e2e245096815fa3f8175f0ab4d3 |
Hashes for blosc2-2.4.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf8b64a7506ba4d834988eb863410bec4ee62c8e523c2ff73b0ee636efd39b3a |
|
MD5 | 6d4d766ed0a18bd7104f803d772cf6f0 |
|
BLAKE2b-256 | f13b99134f6c36e3f6ee20e6102b56e16c09632f5ddd669350abee25f31a910a |
Hashes for blosc2-2.4.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e4529c05a0772bebcf58678b2b0d0465cb836d43c2882c2d82b0b0a6c3b1173 |
|
MD5 | 39eaaa16483ed620a0478cb831dc02d7 |
|
BLAKE2b-256 | 3c4ea0c06c0cccd7f676abf452bcd6f18239c5da7d21fe87e8f3186e181ab3a6 |
Hashes for blosc2-2.4.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92a25def994313098934bc44ada2d7ab8611e2ff0fc2923e3c417cb47ffb97fc |
|
MD5 | 8898ade090494f65cde8cd31cb5f2195 |
|
BLAKE2b-256 | 83b14e57edbd5314c2d007d3669eb37faaf381407e97085f1cd901293c5ad503 |
Hashes for blosc2-2.4.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 497717691e3e3c4253ad574b0405494c68f537056a510cd5f9c419c49a976533 |
|
MD5 | 85ab6df7d3f9a4ee9845b96d0b031ee8 |
|
BLAKE2b-256 | 47b424bbc99daab7ceb1ad7b4f8f7d113881ba78e8bcb9ef64eb5bf09850e8d4 |
Hashes for blosc2-2.4.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdf1c12e29045807ca86e590a1b3f018277fde4af9ac80a7bab7036c52e68169 |
|
MD5 | 1c374aadb0d99c80b6abdb2ff9fed01e |
|
BLAKE2b-256 | 9d1f39aeb4a59af5f5e23e74f906a427d017807099c7550a78449bfbe4853599 |
Hashes for blosc2-2.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d220110b9d3b95667879999a60cb56c3706bee93e5e64b6177c04f84e5a4a49b |
|
MD5 | 9a19000b2dc7e26a0abb395629a10172 |
|
BLAKE2b-256 | 7c51d3d1f86fee0f0b89ebc19e59c2ed42d77b80e6be0f755df7fe33c4263c77 |
Hashes for blosc2-2.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b777159fc9ef5ee45da8a64755f9ac965b642774dab86d89ad12207e9b0fc894 |
|
MD5 | a95ff7160245743a44d1d44bb92332a8 |
|
BLAKE2b-256 | 427372060ced82c0b4e4510c81707ccf44e6f2b62618dc98807ae96deb1d021a |
Hashes for blosc2-2.4.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12c1f4dd9eae6e1f6a95cabd92e0a18063697c2f32329b4dfa8af5d5b075c86c |
|
MD5 | e74b19c7108ccdd4d14ecdce6b76aebc |
|
BLAKE2b-256 | 3957323709682921b96ca0a309fc2f9935a1d89b5795f5a5063c1c80f14af4c7 |
Hashes for blosc2-2.4.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ee3fb14f83e894ef4491b11b1d12b3b374e4d8b1c632d9d0c62c66be5c2c263 |
|
MD5 | 54862154f99ce97c12a5a052ddae76b5 |
|
BLAKE2b-256 | 065f62ef73674f34de7d9503193e3770ba7891e8761cde5cdf94ee6fa71061d7 |
Hashes for blosc2-2.4.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7400095856808b2b5016755a7a771a65d698812459f1d87906598ae53c130b3 |
|
MD5 | c419915f8e77ea1e972a2eb214599f4f |
|
BLAKE2b-256 | d0b57a5a70e6ea72a29039e29b95bb56eaa3a59933067092e353ad9f66ad8350 |
Hashes for blosc2-2.4.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60aaf2988516b0ef8dbc71b73b1a5aa4af18bdb4874fee52d725e81c78d5a562 |
|
MD5 | 486f57f0da55b46ba7d1d20e7d0f1823 |
|
BLAKE2b-256 | a028f69c0218f25743eaa01794b338cde3e5a7c7aae63c260d7f18cdc9e1bd35 |
Hashes for blosc2-2.4.0-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5fbd40e36def428aca963a5aeaaa1874e21ddd80b9c788d753a1f1d182cdb4a5 |
|
MD5 | 9f2fc4beb9e7e52f51438ae3dce4aa4c |
|
BLAKE2b-256 | 65bfcf7756b21da03592170c163ce6fca55b67f10465b6a4463823e16f2605ab |
Hashes for blosc2-2.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2141dd8a04a775f374d12ca6c485208d242c67b12aeb91b025946da1c3cfa5ee |
|
MD5 | 00f94626e11215a37b018437dcb03397 |
|
BLAKE2b-256 | 907d1817e09e3fc83f72d38e12abe416eb5ba0d5aa21ddfa9e3ffd3b5d2caeb0 |
Hashes for blosc2-2.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32fd1d796b8c4692bf57fd2f9ea9159420087e090561e92ab61d429a21a3d5cd |
|
MD5 | a267e7e0f08d9e36c96f4cb81ba8cbd8 |
|
BLAKE2b-256 | d541a740fb738aaa6da61ffe2be49e86ed5398ebc2c763b8d8d1c9e1acb591b5 |
Hashes for blosc2-2.4.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10eda33ae55f5ced9d99fb551728f519ddd028dafc8abd5ae24f40626d6c4f0e |
|
MD5 | a59f8a57fa6bfbabbeb4f68f4127dd46 |
|
BLAKE2b-256 | 4727f2f095cc4dd101c4dcf21469751857f107c460de3beb61d16eb87ed0981e |
Hashes for blosc2-2.4.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36c1edd3e31d446fb1e0dddc6b0cb27b58c79145cb4df74314a502a9cddf92fc |
|
MD5 | 53e47ec892933220a0b17e23d9587e35 |
|
BLAKE2b-256 | cc59ecd33a2d5dcd71237fca3caf91ae7aac40f0622cff0f22ed6d5b99cb610d |