Skip to main content

Python wrapper for the C-Blosc2 library.

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.


Enjoy data!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.1.tar.gz (4.2 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.1-cp311-cp311-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.1-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.1-cp310-cp310-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.1-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.1-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.1-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.1-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.1-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.1-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.1-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.1-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.1.tar.gz.

File metadata

  • Download URL: blosc2-2.2.1.tar.gz
  • Upload date:
  • Size: 4.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.1.tar.gz
Algorithm Hash digest
SHA256 bc01a4a862276a25ab33cde3f99b7109a704ffc95a0974580b180837133aee59
MD5 e7a2182c54b623786dc6d7247f993a2c
BLAKE2b-256 5f5cdca4be2e985daee4160d9eb50771108b2150d4dd4a3d5dea6e515ce26a86

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.1-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 05643c03786b64082b36f40d9b4a7151669494ba439b9765a768105665c00a72
MD5 04d12faabd1fca3c2c55d9ef4f0bb489
BLAKE2b-256 545ded012d730126631011caab51577f235c734278f3e209dd75c63bf27d7d67

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 01062444e6cce13987b32b3f605c1e678c1f77e1ddce91c2e7492d4bea828697
MD5 3a595da95cc58d977c5f0c76377a7d71
BLAKE2b-256 4a5c5a15393fe80503a0b5a0f4c1b880d80d19d8579b217b0103a21f6f070a55

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9c092e2d997001e18119d0ee0bed1ccfb6c1be67473752ce6f6de445d8b0ec16
MD5 dbc5c7aec3a4756a4e3a0653a88ee503
BLAKE2b-256 25998bc182a15bbb1fc93d23beb209e0c1ee8f75da0bf7b32a1f06378f1d08bc

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e4faba91c7ffa9f7b0c595eb0008ac045d0b85c0b213d6f63297a8cca29faf08
MD5 11a7a13bde24eeed9e2870416bd79900
BLAKE2b-256 9ae0d66174f51b22fb8e9ebbcf17d78b87ebf108f517fc51899ea7e171401189

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 5017c709f9c1eae09206bab41249bad141f7a1a1295f8bd16d4d7a40fcbb9773
MD5 c3afda0628174ce318e6d33a6f15953a
BLAKE2b-256 174560263d9027972f80671ab6515a0a4b02a7ce3e850c176dea0c4d0a23b994

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 677b0c6a69860f8ca56070956d7510e5c6c43a7379a6f7320d573c541f58211a
MD5 3aaac83ce599fccb2f00a6ee36bb677a
BLAKE2b-256 43ad2ab01148d488e7b179b1ecfe178e08fda0c4add5c90de0aff80726616a7b

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4824e3263f6462a08df7efa1d3e1de8b04d4c96a2529ecd6278498bc440125e1
MD5 01d31b53d02fb82dd0097d71291de8dd
BLAKE2b-256 82b169d21c8a48a3fa771c490c80c0eafb77c59a6861596f140e8cc7c84bde98

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a22eda1c2723ce97a5395150592b30681e3edc32409b97e0500230101a8ea760
MD5 aa1ebc115a82f82bf39dec35a36845f4
BLAKE2b-256 5292a9cc2926e5361a88363d33f77936dafc95299c4236d66fd45ce4a7ea9472

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8d31434048adc813b9e921ba581d0cb960ca245a4d1ef116342b2bf3d27b9bbb
MD5 62c6b0fc250070cc1a65b70c9cbcdde4
BLAKE2b-256 b29b715f3b09d2e3e5bd390fe53c85bd68ce838277de5f6fa2e12f024eca63a1

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 24dedc40b3e42a1b6b47a72901ab2c9ecffd6047ad4f5344e9041f871804ec2b
MD5 d839e3f276c72eba22b5a06c375fad90
BLAKE2b-256 cb2b38400c0a2c49ba3e1580ddb6fafd93a4c20189bab9c0a28bd334314fde45

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 66fe439152c7ec761b461515167f6226bf78812a88f6eb291c289b6e0342962e
MD5 20d3359759cdf7eda9203af3beacee96
BLAKE2b-256 f390eaf2a677727ae0498ef73858cc83054acc726fa42f63479b79f306f33a05

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b549a80cc52b2fca846d5c9b0fc80a4949b483ca484117e6e95b5ac7f7d874e7
MD5 09b5a20fc08481ba65db5b2abff9c33d
BLAKE2b-256 f99175ac6e90fd648d9f1b59faa38a2edd4fd4b9e07f305738ce9790b0d1f4cf

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 19169c5deff308986e0e7140a90f2b0c48fc7fb54aa41960c66257bc562d3101
MD5 e4ad3431406070acff08cb3fc4ae5c77
BLAKE2b-256 e14468509628134daf1788f91f9907466956f964711520f9666f6707afe3bb60

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 bf1c7a958881a6a1963fe8d79dfb54eee4373bb2bd5c863378edfa1c93f2967b
MD5 c87e158f838064688702022f36552bdb
BLAKE2b-256 d8dbb59a85474a91b77786a7db42d3ed50690c113da7608da569133906db420b

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 1feaa6b931eb4501a608313d26470cc9a7de0578fe463df7b521a124af89f27f
MD5 c9bf6b3b5ff7b00f1615848ca0aaf46d
BLAKE2b-256 659d6e07863f9018ca5c6c95df597b0ca65776c29d533f5b5587b502c25dc7ea

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 8cdc2ffec7fde3ea0b61e0ef02c7dc160e73a33db7274314dfc75e973f3c1262
MD5 e66c3c7e0fc91851809fc3ba41e5f3bb
BLAKE2b-256 133d8e7b15a691d1e3c998785fc791810e4236a584bb3020c67888fc9bd1feaa

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0e299b1877207ebe40b64ed6d341bcc9c6f7cf980cf72b08f6f91e81804c98ed
MD5 92bf331c1a2bfa408a8df11ad20334bf
BLAKE2b-256 ffe1eb2324081ee63dfa94bb80f4ecc28461da69ee6182f3688c12d25a184d57

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 02834e4a545c7937b4dea324b15b7ab0768c05622ab42eab46a3605305ea57f4
MD5 b45f01768a246379092b8f60acef6e1b
BLAKE2b-256 76d5586f2c0ba3fdd1bc099ddaef9f47593a17d503086899c3ed44f75f8d07da

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4f7cbd18bd26ed2241f0c92c98d4a8c5ad2c030efc2e2c47095474f354c6b479
MD5 6e50002a95d96c1df2ed67948103f952
BLAKE2b-256 a17dc82ec9e9ae7c78d0bb60d1dd008c83fa1347922ce125b588e7a75ba3d75f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.1-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.1-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 96fb47707c0b4b38e1a8905c477541cd7903ddae86e4c869bacd810c97813b68
MD5 3fde80372e63ecc4626f5790b2e8a764
BLAKE2b-256 de4ea6ac6f7d12ebb0c5b1af3ab2600049ad509a46823782144808b3c47ad9d3

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page