Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Enjoy data!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file blosc2-2.2.3.tar.gz
.
File metadata
- Download URL: blosc2-2.2.3.tar.gz
- Upload date:
- Size: 4.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ccd4ee21ef1e25ac54119c4a525b6bc2ba2a9a747448de1b27cf8240ce94fff |
|
MD5 | daa2c9bba017e1b26bfca5fad6bd9669 |
|
BLAKE2b-256 | 2137c12af4f7677a49da13444a158e2ef44989a4b6d5cb6b5b35fc17e6d07e8c |
File details
Details for the file blosc2-2.2.3-cp311-cp311-win_amd64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp311-cp311-win_amd64.whl
- Upload date:
- Size: 2.2 MB
- Tags: CPython 3.11, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeb840983f1ed7bcb23526379e618ccc5a7f7c9a62ae58ef532ecd52d95e42ab |
|
MD5 | 24f2b214001511d71710dc5ae17108e7 |
|
BLAKE2b-256 | cfc286f307f490a153ad01edf96deba4f21264f36b3883e2ec853112869a7ab2 |
File details
Details for the file blosc2-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.11, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a74531d2907664120561325267124ff89a374377223170b3cf29439c93c06de |
|
MD5 | f75a9f9765d4f9ea4a9ebb1416a9056b |
|
BLAKE2b-256 | b31cec35f535207f4a0d14b181b90d63506b9126ac2a3af99b986efb69f531ba |
File details
Details for the file blosc2-2.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 4.1 MB
- Tags: CPython 3.11, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77ed06591577dcf6746064b94155886be6d871fb7009e6a9ca50832a84e1f9b0 |
|
MD5 | f5b55c45b8a4a23df5bf9b0484aed438 |
|
BLAKE2b-256 | cc25d110e1887473ed1c5403df06bc0c68d4a44eb12dd6053994d0e32c8f5e34 |
File details
Details for the file blosc2-2.2.3-cp311-cp311-macosx_11_0_arm64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp311-cp311-macosx_11_0_arm64.whl
- Upload date:
- Size: 3.4 MB
- Tags: CPython 3.11, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7bd0a91ac72004e655c67e1a42f1a214fe447d77bdf9ab381c81b35294b5752 |
|
MD5 | 719861fec401db2ec470ea140b595e28 |
|
BLAKE2b-256 | e6e3edebb9dd03bf26fc900241b0b79185924dda858504cca2a2bb9850a410c2 |
File details
Details for the file blosc2-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.11, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a1a798267e6e5c63b2f2c853789b1faf875fe07d8a0fbc50c1cf1e0150dace4 |
|
MD5 | 3c11920897d1c53c44f5e501d0581900 |
|
BLAKE2b-256 | 5bcde5ea9f4167e06e0e4116b9aa76a4717e2e68d453b18d18576ef4f636e587 |
File details
Details for the file blosc2-2.2.3-cp310-cp310-win_amd64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp310-cp310-win_amd64.whl
- Upload date:
- Size: 2.3 MB
- Tags: CPython 3.10, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e63942ab4fd831e37755df55cd665b6acc9b04ba5388eb4922fe34488a1ff883 |
|
MD5 | e31c3a2211d39641fe3e3b5ba6313c9c |
|
BLAKE2b-256 | 0ad0d95357893f4cef8409ca040deafa069bc6af87a9cceffecfe8ff1808be3b |
File details
Details for the file blosc2-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.10, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a418c1a72b9a71ccca9b71c44eebb491b3287b7ed38a82a24684b7f2d38a2c7 |
|
MD5 | 5a9db46fbd30580c763173827f6ed5da |
|
BLAKE2b-256 | 0cadb8ba993078fdc83119081b4290bda160a5b074048169f19785d4091a961a |
File details
Details for the file blosc2-2.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 4.1 MB
- Tags: CPython 3.10, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c33066f1f70431e252c0e0f8c7743112c73d2442e78b63abcf68de779c60f9f9 |
|
MD5 | 3ec7f9b5b2ee8983e6dfbfc66aadcda6 |
|
BLAKE2b-256 | fcf5d2ee6bcfda5627ca82e2cc6db33127f73fe8fa8817ae5251ff3a9956cc64 |
File details
Details for the file blosc2-2.2.3-cp310-cp310-macosx_11_0_arm64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp310-cp310-macosx_11_0_arm64.whl
- Upload date:
- Size: 3.4 MB
- Tags: CPython 3.10, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4828c6886cff6dc51e3cc3e8e21f24ccbc58ff03b481c534a51461e380207105 |
|
MD5 | efe4626ce58a9af2a1c29b6e46c3e1f5 |
|
BLAKE2b-256 | 9654f570d33ca4ec145ef13d84f16305767471b2a5995d0608fc8087cec4146d |
File details
Details for the file blosc2-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.10, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b9bc5dfa19081b2f4ec856181af99fc9c438509e4102370ae0ec9756630fd5f |
|
MD5 | f036eea5c2f467afa92bb6ac7029a604 |
|
BLAKE2b-256 | f0c891b6a42200441e0c632d43b49a5be6e9e3b3715cb84bbf18c7a90d18bd21 |
File details
Details for the file blosc2-2.2.3-cp39-cp39-win_amd64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp39-cp39-win_amd64.whl
- Upload date:
- Size: 2.3 MB
- Tags: CPython 3.9, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf97e16ba60ba0fcee2aff7314888ee7c03d451cafe3f799d322b0e6261dc8ba |
|
MD5 | 848459efe385b98f90b71dfa2a435df6 |
|
BLAKE2b-256 | c4642377162ea17dfe04a277215878e507e6ee158fc74aecb17eb20f0534135c |
File details
Details for the file blosc2-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.9, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e24bddd44402ba91035db594aba0e03d53e65db418f959549ca6dbb5b48200f1 |
|
MD5 | b3db8273865de82605e4ca6bd7366610 |
|
BLAKE2b-256 | c5267572cb5b5fabb137c9b769a73a338892b1df8d0c2060542c56b4090405b9 |
File details
Details for the file blosc2-2.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 4.2 MB
- Tags: CPython 3.9, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a394c6ff4d6e7b10c60a50e1ce94e2fcf8be9b8caa6124db5983bd107e94ae4 |
|
MD5 | 94b8c23dc0ab1647ce948f426e03dd48 |
|
BLAKE2b-256 | cb677f7231497f81db93d610c9c1da3060fdfe9dcedbccea601c5a1b27a0074c |
File details
Details for the file blosc2-2.2.3-cp39-cp39-macosx_11_0_arm64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp39-cp39-macosx_11_0_arm64.whl
- Upload date:
- Size: 3.4 MB
- Tags: CPython 3.9, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4902026d44826c0b725963098e1a3c29abfb679c776a0f2f875488316715f8a |
|
MD5 | e6ec321819ef1d451f3fc9ebb9db0476 |
|
BLAKE2b-256 | 39e666e40e0b7cb2bb66b3f09e72a10af3f2a9ce2b045063e4c0977e4f9d81b2 |
File details
Details for the file blosc2-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.9, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6078dc80cac9aa59e3d8aad2b4ef88b8e798c4f5531fa11d613aaa7945920821 |
|
MD5 | 98d4963d7948400ac88ba377c3eb628b |
|
BLAKE2b-256 | 894bf7c8ca526fe10b54384007a612138e52e26b49d27ef53684baf3bcc05300 |
File details
Details for the file blosc2-2.2.3-cp38-cp38-win_amd64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp38-cp38-win_amd64.whl
- Upload date:
- Size: 2.3 MB
- Tags: CPython 3.8, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddacf096140e515032cb97a0512189183a57a564acc356a8299669bbb2905619 |
|
MD5 | 3b8793ee9b282492e43a67d94fc58d32 |
|
BLAKE2b-256 | 5c9e04ff2aa8f1e38f5fd7fc6b33eeafc6be2f435863b6f1af7f5383cb4b4e02 |
File details
Details for the file blosc2-2.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.8, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b470af7e42e5fe1b122a4859086a7447c49808c36c355dc2ed4121b8be511f6d |
|
MD5 | 8631967482636bd26e40aae37face1a3 |
|
BLAKE2b-256 | c1390e95cff858b2c97540c7d0f159da6550e1503cca0dbe0178f5076d104d12 |
File details
Details for the file blosc2-2.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 4.2 MB
- Tags: CPython 3.8, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30c32a36ce83a37d6e2c14c1eda4ea0382f4358bcacfc243a7fa39a54661ade8 |
|
MD5 | 26db2f67b3bc811960b7b93f1f140a4d |
|
BLAKE2b-256 | 0500f29ebb5f376f305e063f0357240043c80696860fd88317658133f5b6cf0e |
File details
Details for the file blosc2-2.2.3-cp38-cp38-macosx_11_0_arm64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp38-cp38-macosx_11_0_arm64.whl
- Upload date:
- Size: 3.4 MB
- Tags: CPython 3.8, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e5cde5b5f2a4bd92e0ff6d381e89b367e3cb61c702e01ab264536339592d6f7 |
|
MD5 | b18c0c712fa7070eb3810b483330d23e |
|
BLAKE2b-256 | 8f9817306c38676b4d8a6a789fced9a4765e24c8ef3e546a5a9c4a0fc8c07f74 |
File details
Details for the file blosc2-2.2.3-cp38-cp38-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: blosc2-2.2.3-cp38-cp38-macosx_10_9_x86_64.whl
- Upload date:
- Size: 4.3 MB
- Tags: CPython 3.8, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.11.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0020580e4899d7c7573acb92a06d895f783e9033ebc53991d458e7ac7515ed04 |
|
MD5 | be5d880113da41f57a8bddab833d241b |
|
BLAKE2b-256 | 77d2ded29ede9927e3371a5dfbc002a2a9fe6c604b47b654b11debd2e0acf490 |