Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.7.1.tar.gz (5.1 MB view details)

Uploaded Source

Built Distributions

blosc2-2.7.1-cp312-cp312-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.12 Windows x86-64

blosc2-2.7.1-cp312-cp312-win32.whl (2.0 MB view details)

Uploaded CPython 3.12 Windows x86

blosc2-2.7.1-cp312-cp312-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.12 musllinux: musl 1.2+ x86-64

blosc2-2.7.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

blosc2-2.7.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

blosc2-2.7.1-cp312-cp312-macosx_11_0_arm64.whl (3.3 MB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

blosc2-2.7.1-cp312-cp312-macosx_10_9_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.12 macOS 10.9+ x86-64

blosc2-2.7.1-cp311-cp311-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.7.1-cp311-cp311-win32.whl (2.0 MB view details)

Uploaded CPython 3.11 Windows x86

blosc2-2.7.1-cp311-cp311-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.2+ x86-64

blosc2-2.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.7.1-cp311-cp311-macosx_11_0_arm64.whl (3.3 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.7.1-cp311-cp311-macosx_10_9_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.7.1-cp310-cp310-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.7.1-cp310-cp310-win32.whl (2.0 MB view details)

Uploaded CPython 3.10 Windows x86

blosc2-2.7.1-cp310-cp310-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.2+ x86-64

blosc2-2.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.7.1-cp310-cp310-macosx_11_0_arm64.whl (3.3 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.7.1-cp310-cp310-macosx_10_9_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.7.1.tar.gz.

File metadata

  • Download URL: blosc2-2.7.1.tar.gz
  • Upload date:
  • Size: 5.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1.tar.gz
Algorithm Hash digest
SHA256 34db4d41543b16e894d507d6c7d10a9a67c9dd420ff40b072f7c39185cbb8fe8
MD5 625c862dcb47ad4a0b70384172e7769d
BLAKE2b-256 efbb19a5d672f86dd26be0fc4f3a4c04264c088f3309b7b9d4e3e853a1f3cfda

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.7.1-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 a1a884f181a2a9612470bcd589794c7adda3eebdaed918e1a8b6e69683524b70
MD5 41e06e3f8d713edab966a9bd5166b71c
BLAKE2b-256 778144def26a468e14309a4cb7d5f6f0534e3e9eec292c0453a9624a4d9c0ebd

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-win32.whl.

File metadata

  • Download URL: blosc2-2.7.1-cp312-cp312-win32.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.12, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-win32.whl
Algorithm Hash digest
SHA256 6c894ef42fabc49c6bd1c41d2e45ac24e911b3b711c1e26f9be93dbcbf5fd4e2
MD5 8c14e36996a02faf0ecc6d9e35f5df86
BLAKE2b-256 9bf63864933ef0d51845a78f260dce9eba4e3938ff136d1dcd7a4b9d91d25191

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 23763b3602df9b0e63d16d21abadaac426a74370a0a986f08bf5322bcfd207c6
MD5 4af67d98c635d3f2097d50955f5b7241
BLAKE2b-256 402b9fb84e2656f09ac90fed163490c11e88a8f5306e53f27b9eb324a2d966b9

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6c1f3e5ed25930596492dddac8022a729d538efb3c1c4e4904a0021288cf4835
MD5 005fd8097ddd0fe474f2622cbf96de80
BLAKE2b-256 589f54099025ce55a33d2be80eda16821d695c0249c9826e14196e2655ca4a7e

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c662a3cab4760c194407de1990bd70608af29ee30d6471072463cdc8cc6586c4
MD5 ec3b85dc616b6c29cd273a6ef9e70ea1
BLAKE2b-256 010035bac426ec7e1e0ea05193043bb2c1a3151861a2f5519ab070223ca5819e

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2c7dbdd93378d01b048536b2eccbb77eaa73054665bc3a9396e4e391a50c4ad0
MD5 f66ac6962ced6e06e3dc06e8dbf729ed
BLAKE2b-256 f1853f817bcd5f665625a46056ea7acad8a4798238a37d85a22cc8294e885e89

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp312-cp312-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 bd91a5d9e1d35b47bf5a3b6b440bc120f1fa0af45264c0a6ccca6b1083de43e9
MD5 ba01b298d46a712373cc2085a368f333
BLAKE2b-256 d883417cc99eb9c46a295c9b9381834e9728ba1c583ad942610755a5348f9965

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.7.1-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 9b00bcaeaf125491cc6dec5508389f95152de69cc678a63a0310a756511f3d21
MD5 e97944543c97633a4ea68c453a0ebb9c
BLAKE2b-256 dfe136db3625759c8e620ed992450e9579a02f023538292883e04189845d98d0

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-win32.whl.

File metadata

  • Download URL: blosc2-2.7.1-cp311-cp311-win32.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.11, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-win32.whl
Algorithm Hash digest
SHA256 bf55816beaf34eaba021904ee7d6c428ca338d0f3ec7a5be04dda552258d0a8b
MD5 18dea37c4b404d96d79d276654d7e3d6
BLAKE2b-256 d8f6aeb8cd6c0184870ad2be3cf3664a23e45595024b47cc036e83f20b2d039f

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 7a369a72467b54fb45aa307cf6e3ae84ec0212a9b55cfa5f4d4eb06a06bb1472
MD5 6447e6813c5aba103f56e26e2b2f405c
BLAKE2b-256 ba4d0295f619fad024e05a46afc1315661f08f3ac3f2c051c9bf060cfb91fe2d

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ed15f0eb6a99df425b33be948fb309a97e76bda37eae20022bc9611bce775592
MD5 0c29412682a411135301b90929a8cba7
BLAKE2b-256 3e0021f408b3257fff4e5166dbd666ee2739f088e139dc90743b09952ee0ebd8

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3c10a9505c6d2d5e58f03a1139b4a4b234511baa9ebeba17f8f1cc22881ca764
MD5 c74172765fb0e8fcf58808d7ef0fba8b
BLAKE2b-256 e093c1051b42f9797504c11d28185b7e4e343e716eb12152f087cf5312aa7b8b

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f8134aaaf72b670f2ddf258d9b64646c262d6b1b112a530d39f9a5d316f3b306
MD5 e20403e099bcc2eeeb0360c630f2f122
BLAKE2b-256 773d1dedc6be1a730d4ba75c599fa8f5426efd02cef7722ed7e54b16e0ae8702

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 51fa45187639cd1f7822355c91f6ceb0fbff6157eb6f9c32462782551fc61283
MD5 d17f514dfac471e9286695dc5df06751
BLAKE2b-256 22cb2a5b2024afa43f9b90a21022c44211ba064dc3501e99e3a8ea5b275189dd

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.7.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 150030e81bd5037f802ce7844281a37ff67e0998aa1735c48b02f3c6401bf0a1
MD5 ce27019de338678d67f3b452bbc14d76
BLAKE2b-256 9ba50bf10f282853aaf77f8f7b919ca79813b56b2d7955625400d527eb268449

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-win32.whl.

File metadata

  • Download URL: blosc2-2.7.1-cp310-cp310-win32.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-win32.whl
Algorithm Hash digest
SHA256 d5e03ec783841c82ba6704638f5847a80cac8625186932a8f3b47bc5fbda4f7e
MD5 680ccc487f24ab32cfd41b0a6284e886
BLAKE2b-256 db646e11754d7498aa9e7fa13436e9eef00aaf1c74a9c67f74b36e134bd2c81a

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 06e0933855a6cba347692f08de1aaa51a8858dda67e7c5eaeaab59663b897879
MD5 2c03fd10e8c830dc0984089a616f36c7
BLAKE2b-256 44fb0bc4828c931402d711e9e26aa9938b7ed4f3abc4e7714edc9ab593ab83c8

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3412e83e6b2508f3da0b0f3e0bf86bef4dac56f018525d2ab55ca50712a2b91a
MD5 ab293e3fbf64548f89549ccedd9121f0
BLAKE2b-256 bb1b5abcd413e11222411ea6032be42aa4db3eac51ab04b967f7e8c669fdb857

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 18770f75174a3aae5712f53e9cb5885b258b730b281cbb3bf1e7a74ea3d94fae
MD5 9098f6268ec802246a586ef83defd32c
BLAKE2b-256 456adc70194a05e03904a5c9fec9241838b3168afba98f31a0cc7b9a4451f033

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5923a22fb7cfdad571b316930e8812618ba8b2ac1f37ac05edcf3eda71beed9f
MD5 baf9c81a354e39cb071bde32773eea92
BLAKE2b-256 3fe70f259887dc4b5ac48ed088a932444052905aba1a883d2bd79fd54f6a8ead

See more details on using hashes here.

File details

Details for the file blosc2-2.7.1-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 ad698c6fca8dee677b7a1881206f7ac43ffd7c458733716a3de1e1f3c6ba1f79
MD5 36e28868193821efd2e136657db5a88b
BLAKE2b-256 93f4cbd6c3aaa6d328f73f6c86d56df016edb48083291b90eb945572584fa8f6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page