Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.7.tar.gz (4.7 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.7-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.7-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.7-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.7-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.7-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.7-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.7-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.7-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.7-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.7-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.7-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.7-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.7.tar.gz.

File metadata

  • Download URL: blosc2-2.2.7.tar.gz
  • Upload date:
  • Size: 4.7 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for blosc2-2.2.7.tar.gz
Algorithm Hash digest
SHA256 7b6d805446c560b26003d1f907e7bd368edd82d232833e9f3b617e2c88108d5e
MD5 1d089f80825582015b021cbe0b66f205
BLAKE2b-256 4462af9c2fed897702ae6881ec77ee15dbaeaea14e2dfaf843f854121ab8505a

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.7-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for blosc2-2.2.7-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 fb1b6b6a46e3246e55eb69b438a61fca2428db3bbac3b0d5335b23bae7bfdaeb
MD5 75e9f3c46528658033786d7ec63c7c68
BLAKE2b-256 e4a08e04e2a2da84af69b7c7a8676eeb67a01a9792c19996f28a30dde63566c4

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fc21e1024856160e14196b9dc61dc3f8a38bf6572ed0abb656f83d321359d130
MD5 b11477b4e6b408b1f0ac76eba549acd3
BLAKE2b-256 d3fc694d352e8bb4d66ff2b69390fb7fce1d4d647a1b3d9e696c4d1a55339951

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e69e7556bab20342db30e9fdfd7c198b12c8c5306c4b4831702b62efcec5298f
MD5 aab9991d5dd374d2b2e8f36572d69527
BLAKE2b-256 9d1e6e0c01d76aa2b5a7a2e8dc63ae423f8427765198affe182abc26b64d44f5

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 676d38201087aaae953701e2006d331c3232a7b091a78077d9c3a794a51c10ff
MD5 ee6bf67768b04b5c5b5221c0be0b777f
BLAKE2b-256 393219508648443c7d97863f5df2f7dd900a45558a7863b3ead227d0e6091abe

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 76683b4da807cb8e5add36f9098f89bfc8e928758e1a5ae2319e952cc014bd08
MD5 41dceeffa1d8c717b247b1673d121ec8
BLAKE2b-256 5b0f0161b82665eb8d450943a7476bad30518048f0a3518f25e736e5727b39d4

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.7-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for blosc2-2.2.7-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 25effec18d3a2a30dfc6b828735141a806be049d6979aeae34e922ab42ecc288
MD5 8a9ed85518a474183747bacd4e49fc36
BLAKE2b-256 50167e7e01ff544079aa85e102be30f1b04e6c49497b02ef5ff47a628f02f5b2

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 175cb2ea313cd1676952519cf33fbc9931356cb0b4df61894c82a152aaffc6da
MD5 7b6d4aeb67b5dbc3054ae83cc9fc80d0
BLAKE2b-256 425cb4258c15c7a5333a67b9601368235b6cf2aac1cbff40cbc5f707d34af611

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ab5e43ea61794c59af072aac4adeed4bef9806881c94cb3181da72870ef77528
MD5 533b026ee4669f51cf603d19eda6f448
BLAKE2b-256 ac1f2d34a917cdd4e1aa4ba068cc533c1da134b462ac5f20caaec8008f113974

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 73c635be41b0a04e290c85b2c3b1a06bd4af7bbf9556686f0b00c7306e3a376f
MD5 9a64f8293c967dcfb501bb24350a7d73
BLAKE2b-256 e9f91d6f0d2dc70499e8eb7a389fea31898c5c834296412c5a17e083fc58bb99

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 b089c24831f23b49e491fd3c57d8b4ca2921739b55816b68905fd1e5d173334b
MD5 8ea29540b6408264033d467051f48160
BLAKE2b-256 247752fb923c9e78edd42132cfd67a4f639491efe3c87032c552d6962fa91313

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.7-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for blosc2-2.2.7-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 05c7eafd31598072797b92f7de67ae575b57caa5879b75d699e8c138517aee22
MD5 988bac8577ff2ecd337dc0e4f63e5af7
BLAKE2b-256 ebaeb1683ff2af362b7ac7298861526ff38f0d663243ae7650029b4844a5e33d

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 be0e0e5bf5fa458af42aaa76deb5e901da751ead215448c2b3d2f046005cf88d
MD5 283d3461f567bb73d6a199e57430c96d
BLAKE2b-256 5410afe42d81959a2d8843481f2e20caca378af08ca4629e129c608bd5120d49

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8cdc1017a7f97769b7c4cfb772294aba0fdeb894efa8bf31d68b0ad2b859169b
MD5 d83aef727da25b96e649e6b4e7117835
BLAKE2b-256 bce8a92d462219284ddf767362c710b8cc4a52b351b29fbbf6ebab95e4e10355

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9498db576a402b77cd5e89e56cb58c7f45bc20eb0f482612cdfc3025baa31961
MD5 bb9a9316c1f0d23b612409bc7eace43a
BLAKE2b-256 a3dd968ea5f3c5f9cc552b7734416e3a4318620c1da1de874c796d2202147f1e

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 85c76f7025a325831f0610c2848cee46a88fe5dc89813e62d00b3bc8a28c3fed
MD5 16986b7a9bea390bde437fc9949bcca5
BLAKE2b-256 0a4840426df65eaf48285404230e22b3baa701d2c89bbb6aa1248fc74c7f2fa1

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.7-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for blosc2-2.2.7-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1e165221bd7982786154f7b6acc30dd6b343ec4d8d15a66871306dd14acfa835
MD5 3852ccd0ffe2d2beed9a33b92ee2a771
BLAKE2b-256 23be2e71b7f66fcb3e287bf0d3863bb37b5a59e12b3b97b4cdbdff292785e0a7

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 32417cd414e422257fefbc1846ee7f5293cf69a88f3210ff365d296cb806b941
MD5 821e13ede7837a1ee24df104fee12c44
BLAKE2b-256 037089c0ff5c44ac10eccf7a0257810e08dbee6e4d5956ce547ceec1398519cf

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 771215bd16f333624d99a2deeaca4d9f4d8be55b26f1dc6ca71e8c3795d2da07
MD5 a7e3e4bee046c32326c9558e3bebb323
BLAKE2b-256 4a2d0525f1b7b3bd3bac31bc245cbfe3626e2b2cad26bb93e4876a53f52c70ca

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8f030789157b95bccb48a4ccd6e3f356ba529fb3470f39d62378ce6bbf6a165b
MD5 e2f9784b5230120204f9223d10ec567a
BLAKE2b-256 f9781c2330199b2997ed8839c1566301931f0243163d9ca8ddd81500d40e8428

See more details on using hashes here.

File details

Details for the file blosc2-2.2.7-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.7-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 95234dc7c295b077e7e2e9324ca43c319c847dc453ec7d873d61a92d65568b37
MD5 e4008cd8d2b1b38e9834ab7020233a6b
BLAKE2b-256 ca169ccaf5398f642e3b929adc96888a259ebc2d4628e35c5dae0974b919ed1f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page