Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.8-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fcdfd65733c2b01b04ca137fff49633382f16af3b558c30a05caf88b43aa3a2 |
|
MD5 | 6ab67f8404941fe81ffb8251893b1665 |
|
BLAKE2b-256 | 1ef9e875197daa5978bdb3f8d47073bb110a20974509b8ff2c082c48deec2d1c |
Hashes for blosc2-2.2.8-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c85987729594a4b421e337f7c86c7d79bb58c3b5ed64f5bdabb201fc9b5aba2 |
|
MD5 | 29437e7f32a8d7a395ff81a7bb7f2683 |
|
BLAKE2b-256 | 0f6f38198a2a2faf88c9fd93ce34590f3ddaf62c672e53d49f14f121baf42d21 |
Hashes for blosc2-2.2.8-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce4e6264ac419239e91df4fb73000307e4a6160793d943e24bbd3baf6803dec6 |
|
MD5 | 564eebee570212e9db4c4438470c80c5 |
|
BLAKE2b-256 | d3b72dbda7cf973b2914d8dc8da5cab0c9df4c81e88530a1ea234e454278bc57 |
Hashes for blosc2-2.2.8-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 092c3e10dc15199ca41f95d9546ee35c93448a28a42a1977306b2afbacf9819a |
|
MD5 | e8aee4bf379d8d28a9f8440362cf95cf |
|
BLAKE2b-256 | 5613593e5bd06ba3a2df0841e68c85f56df8046556fc264b0d2b299f46d117e1 |
Hashes for blosc2-2.2.8-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d7e194615c0a54d69f51af3b0cc2ce3c5c09a1186325167e76ab9a0bc1441b8 |
|
MD5 | b170f82aa3ccf23e0915f0aebc5ff957 |
|
BLAKE2b-256 | 895a0339ddd502d0424e24f07595ae7919a3d4082c669041d3b6d44aaa609283 |
Hashes for blosc2-2.2.8-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6d7784b8528b605328dc3129a0f0a550cfa5c3394c6708f813cf719ca40663b |
|
MD5 | 3385250be5e443890638606112f475b2 |
|
BLAKE2b-256 | 8a58625171519d4258b21692697580c6de2a2b0c484368cded20ace5e29499d6 |
Hashes for blosc2-2.2.8-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d60b622c39bafb6b2ea2cc21646f0ccfbecb87eafb95992aba8d2098c573620a |
|
MD5 | 67bed499ec51c0d06c53b8df0625cb9b |
|
BLAKE2b-256 | ec4b63bf0ecadc8697b9c8e58fe81c300e8929278b54999d78112cfe5f30a6a2 |
Hashes for blosc2-2.2.8-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 624a0e8c4e50b858845ebf18ae24ef793a7fd09b4f7955c033f92b70f8100831 |
|
MD5 | 41e1be0bc8e5ff08fb530b7ec81b8620 |
|
BLAKE2b-256 | f09f4429256b9a6a7078906cf46483967ee7a69c70bd57d7feae5a2403a0e5db |
Hashes for blosc2-2.2.8-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c770a710cc8bd4094960206ec4746cb8b12f642d7027093f0a346d822ee966cc |
|
MD5 | 3f593269020636df882632f632cc9f80 |
|
BLAKE2b-256 | ffbb60aeeb3cb07d055eeb0e439691dfb327da1e317e2b371f3226fac4424647 |
Hashes for blosc2-2.2.8-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8511d755aaa43920b0303c8994541b61a0c00ab5e2565f54056571564b1c835 |
|
MD5 | ebb945b100baee719dfb0f7d5384d500 |
|
BLAKE2b-256 | 3c0d39aa590440dba27974167eecf34d7604513e165684393eb4e12954e0175a |
Hashes for blosc2-2.2.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e827795b54b2624e0768246b2661ca3e860c98dac6440dee46833184e4a1b4a9 |
|
MD5 | 72d1cf2fe6bf2561030b87df5ecfec7b |
|
BLAKE2b-256 | a222072c4f297b74ea47fead3648ac498688e2e501f9babaac22af1bff3bea5a |
Hashes for blosc2-2.2.8-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a3d60372a0aaf2409d9952cfb061a4d5a2afd07a06ed8332591d325d1703475 |
|
MD5 | 490df27bfb905e93bda41007cefd2fee |
|
BLAKE2b-256 | d4b514f7daec95cc75d6a7ab68c951187600545aa44827dd80ac7f6a4c490e98 |
Hashes for blosc2-2.2.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe47cc9d570c8cbc1d475c95077b156afc7dd35b1fdce042323e23364d99bb6b |
|
MD5 | 2619f9d5da89078e96d13abf8f2c7415 |
|
BLAKE2b-256 | 4b72528e72df815a59ee7b75c1fb4ab5efb710f9f144b017c093acf2d675171d |
Hashes for blosc2-2.2.8-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e0d9995e088555fb7f347e57b64c337971451c3b086e279cb56e7081321f2c4 |
|
MD5 | 8a75c7512bb5305fd5559d819a0becff |
|
BLAKE2b-256 | 8187d491680097ec1a8d26dedfae896a6e5ee6048c00b7034c2f816a8376a5e0 |
Hashes for blosc2-2.2.8-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4423b8adea627cbe0b8da8a796436f8645ecc5c0e410d87c4fd2054ce051e14 |
|
MD5 | 9248e3e4bbaa7624b0d55db7c13cecc9 |
|
BLAKE2b-256 | 7e7fb22fbf2d3f6352ad1cece47a5d7d8bb680058b3b286ad0e16c35b3da7324 |
Hashes for blosc2-2.2.8-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae38c060e0206a5073be8c60f3256898f938f5e033bca53049e7065cdf4660ce |
|
MD5 | e615b17a3e264407854057e860e14373 |
|
BLAKE2b-256 | 16980099313a810ee60354f2cfb1fd8950fe027552dbd88347d757c54a1a85c1 |
Hashes for blosc2-2.2.8-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2c0c0d6aa98d3a42de51cad22e17832d468d4ce2c30016c0e6aa8e36988459f |
|
MD5 | db92507f394b0325c38515e1be8d3b76 |
|
BLAKE2b-256 | 5be562dba36dd3129a33f574afa8bb0c30f76c1288e0d6a0e50d4086299ea266 |
Hashes for blosc2-2.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62b5dfe5507cd27abaee152c4c5d95c56736259a2b8dfdb74c0e96fd1ad03f1e |
|
MD5 | aeca8301916a532004a43cc236bf843f |
|
BLAKE2b-256 | 2753dbe9a51225c98a8c91c590daae553f978ed49f661df750d10f57a06bb0b5 |
Hashes for blosc2-2.2.8-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c9e78bdb8c5302923bb433092289feae40de67c8d0e5956fbb0d3d037b194ec |
|
MD5 | dea8153484062b162b5ba1afba771fef |
|
BLAKE2b-256 | 8c30e0067a366694c530bc6b4bfa40fc3e794b9bfbe208201b36b210c6e9b8fe |
Hashes for blosc2-2.2.8-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44e6ec246f76bbc2656451a08909c6874bcdacf578d64a8ebe5c6cdcb82d9f33 |
|
MD5 | dc5cbb70ec1510ef415848a37213edf1 |
|
BLAKE2b-256 | d9cd2cf4c01fc2bedea7c2b47b22acfd9c2bc6057d090a8776a9dd79f149891e |
Hashes for blosc2-2.2.8-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c04a6e89a5b45c9a656371dca50e31e9b266d84c542c6c9a73863f3135679f33 |
|
MD5 | 8c5372359a0efd3349723a63cd2b8f3d |
|
BLAKE2b-256 | 642a28ebe49090f83cffe7b2e24a45e730b6561ec4b9a063ed3b924a267db49c |
Hashes for blosc2-2.2.8-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8aa80404f78b8ec79dc535bfb694a338669b90238e4d0f2587787724e19ec1b9 |
|
MD5 | 139067366e201dbe4cc191c210b65cbf |
|
BLAKE2b-256 | 628c3521dc7c399d8dd7503c0ac19f75df8e63bb5f3eb281e15368e067d86883 |
Hashes for blosc2-2.2.8-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 395b9dc693e8e35d8b22ea4df2a1b73ed85a3820bd5b44d706156314f9a6ec02 |
|
MD5 | 935b7742062bcf748f287689315af7b5 |
|
BLAKE2b-256 | b4fd494f2c46a159fa8b30361833be14caad4c804d58b86baa38962ddf413104 |
Hashes for blosc2-2.2.8-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 463f57a8c95e71763efd516322d3e570ae1dbf0a2741238b4291e9166089e6f8 |
|
MD5 | 0417c8fc8b658b2a688047b267e61d4f |
|
BLAKE2b-256 | 541e45f99eaa9753442430f6dba51f4d883a63c79000c1cbab29d422700fe6a3 |
Hashes for blosc2-2.2.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a2102b82254c2572f5ef8951690598939d91c011584053450cdee75e9e3c316 |
|
MD5 | caa0689c13fd4ee55c8a54e992ea50e0 |
|
BLAKE2b-256 | 3dcb8f271cde48bd5fb4a5b4ef2345e97d374538e62ed66ee3dc5aad65ad06b1 |
Hashes for blosc2-2.2.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6228371f5c29c2858f8f146362c1cdce709cfcfd80e3d4ed32cc7aa57a18820c |
|
MD5 | 25453f313ab464f440d9cb0d0d069fb4 |
|
BLAKE2b-256 | 7cf2874b615a0c820ac22bbad4650fe77a5282fa12ff670e1992435e865f13cb |
Hashes for blosc2-2.2.8-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8c4163ad654b620d1f3b4bc3e5992b4aebbe0cb8887cce07a8d77ece0bc6098 |
|
MD5 | 4f1c5b7f04525a979d980982703465c9 |
|
BLAKE2b-256 | ad8a7e643cdf1528fcb337f2ddf78aa280724341cb8b47f58deb34c98d6e2e69 |
Hashes for blosc2-2.2.8-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 751aebf44ed6ca2525245eaf4b233521c017ccec2203443f4ec39051143501af |
|
MD5 | d87b15b465df45ded9c8a8713796eebe |
|
BLAKE2b-256 | aab6cfee17945a44659b130498eaa63f3d3634449e69ad22024fcbd1db59f49a |