Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.


Enjoy data!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.2.tar.gz (4.2 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.2-cp311-cp311-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.2-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.2-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.2-cp310-cp310-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.2-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.2-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.2-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.2-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.2-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.2-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.2-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.2-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.2.tar.gz.

File metadata

  • Download URL: blosc2-2.2.2.tar.gz
  • Upload date:
  • Size: 4.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.2.tar.gz
Algorithm Hash digest
SHA256 110a00d95f9fb9c2753d7ae01b7cf50c12ed28e67c41aed726f547d5202464f2
MD5 ee8c30024cd63e6cee6f35e8cc68c93b
BLAKE2b-256 d65367a180da420034cb36e27a1b15685fe7539d10f9a56c04805febd4ccc018

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 a3ad209bf035b330184b65168c6f594fb2bf01cadfd0f0ffc0d6c680d960bcf4
MD5 d787d54748194b21171a676e884ae454
BLAKE2b-256 d4be494e0785fe51d4609b2fbb9922811f3d2957b04c12a35818428e67615f32

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 11c342e402e05f5c85856b98dd3e761dad2ea226139871e5e61ef8cf3dede434
MD5 ca9fe63212bd1d25963a20aaf8a5648e
BLAKE2b-256 7d08776f9b589b262a1f8bd13eb4fc37a72bcb59c0a982b0f362f59302ec4398

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 31ddfc4ce288f54dddabc7494deb4005db33d0c702238e01b996403fd0844ec8
MD5 8cfb562954507b733e31983be4953ea4
BLAKE2b-256 8b56b86951d514c6506f79d7abdf0bc7fbd8defb18377dedf2f21ab86fdfd6bd

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 031821a93b4b0c946a555d0cbc666b38d0c4b8f5620ae490a370253b868f7299
MD5 1433f982556b104221b62ad7d46b058c
BLAKE2b-256 3d385fcecfba1655e078f5783674c5f8f8766e07dc3a2d252162e3f61fb9651f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 7923315dda478a2e163b89b8be521211276fbed066918995c3b4288776473ef3
MD5 db599103b302e0012dcad06e250846fb
BLAKE2b-256 ae2578ae5a87a126f6ace15649e7f9a70ce75552a7af32ffb604957775ec1dfb

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.2-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.2-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 10a98909ca1fb0b3b27c1a1b8f7bf6692383e72a21b7bf07ffbb843d7fb76671
MD5 c51b6d2916968a62c8f6d3d1698dd336
BLAKE2b-256 a9cf1d36294c87ebdfe1a2f7d6de2cde55fe0684f89e2874e11e7f65bf6208aa

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d44ff6cc7613286326682fbda95f1383cb35d348247d57b1ca542e172fde842c
MD5 7ffd05fe805d57622da222b62a95c655
BLAKE2b-256 b1758f7e0508be21a3bcfda3bf6386b2a15a3a125eff3cbc4652fdd8e47af242

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c10da5a03e7bc5a5c4d0bdc6d3d7f7b738d79d99f58e3e02dab40c7d1f46a000
MD5 429e6d0d757dce2c2822f815b3d860ca
BLAKE2b-256 3a901043797001acb21274e65519cd3924f6d115ecfbc9b011f3f88f55a58d4c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a096d562036ca4925a815be1c889af2d7dc5d1473a435ac7ffe14a22a31bbaf7
MD5 aa62e8274560643db9899012d2f0e84c
BLAKE2b-256 1abb405b28047c895a9490140250b99f3592fa1db460fb2e1bb3ae2867d2906f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 6b9179bcac964664fdec49fb5a19082a8d470fad9a9c5f34c8ae0b35b232abfb
MD5 3d2603fc99589f1d996ede3cd87825fd
BLAKE2b-256 11288d05616c385ffc4a51121799841fd4aca354328946b57eeea3822a138c1e

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 bf4996834d4730b8e7bc9f944910e76a30d8cd715348fa81389297b2a5ddf403
MD5 bf74cb9f296076d4121ac6601aef8fe3
BLAKE2b-256 063fd69cbd37e8b378afe6bfb2acde80ad0f3e8bbc70d53fe8b6225d006f898d

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1fb575e37aee0a6bb9b2f775b0ef7bb03e18038baea3aec143fdde0f7766b16a
MD5 aa8c6b3bfaf88ab644b220c5b57ab112
BLAKE2b-256 1ff2a68e898699675c64bf0323768456c7058cc83ba310d766cf410cf1f016df

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 da2b29ab3ecda3094d516c5746bf3cf19759aa353f571955620a1ed5c9aa48a0
MD5 930f197436a75166da2baeae1f06fdeb
BLAKE2b-256 543e11f12c403c8d3571c33e41f161e0cd64579ae744f36056b073bf11fca90c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 10513e34e91020ea9eb7677f041fbbb79fba341340a05d232a05242bfbfca153
MD5 8d1b672c74fb01a0890933b063396078
BLAKE2b-256 c76f030a1616d16d57ff35008779644fabb545fb4913e64bad66d7d5fde74d3f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 d392c52dc27720ebb7bb6f54a2c35f2474c9969c6801cadff05b9c9a30103406
MD5 92c9e50fd60072dced76b641ffd372d5
BLAKE2b-256 fb417847a2b5b3184af1270003b15062a40d691d879985caa51bae2d5e51b838

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 4fde2834fb4aafe1a3fce75a5f97604887b2607a3579eae9bea5acfb04abb0b0
MD5 1fe02b49d12fd05e09da7ec9b4f5a00f
BLAKE2b-256 bd5a30244a9bfd42520df2e8a04561bd492ebe2c873a21813cd43f13dca807de

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6356aceae0e560f43f650441b68b89698ace61957d9f1a6644475f9503735261
MD5 21a1a65773922a3f2bc93812010a0bff
BLAKE2b-256 570e3e066e802f6845a3a006af0529a0828e44e31d2cd4b97b1ffe8fafb77c48

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 113b68c7bb37c1ad458de8a0b3c6fac933b4f505bbb693d3854aaa43f03a4ac8
MD5 a83351540ab1f150549b9d6613f98789
BLAKE2b-256 cd95ec9df6ef4fde4dffc1fc0ffdd8faca21bf5dec40a36d5b9bdc59b41d7567

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e605bd2a33e1e206a987ebf7328c09d95baf378f8cc0b19c771779328e2f3c58
MD5 a80f23536280abfede704f3b6a32a09d
BLAKE2b-256 3bfbeb7d355b960b1cf0282f9cb5851cb4b1d4dedd2b369854c5337ed6526b2a

See more details on using hashes here.

File details

Details for the file blosc2-2.2.2-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.2-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 f881f6d053ac46d387e7c44adf68e7337217c7e0de0a987bc338333f9dc08abc
MD5 64a48ba2b89ffa1622894874fcae326d
BLAKE2b-256 1b3d6fd41844b58aebcca5366de54c4e408cfa8c98d1e66c118c4dd49969969a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page