Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.6.tar.gz (4.7 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.6-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.6-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.6-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.6-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.6-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.6-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.6-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.6-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.6-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.6-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.6-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.6.tar.gz.

File metadata

  • Download URL: blosc2-2.2.6.tar.gz
  • Upload date:
  • Size: 4.7 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.6.tar.gz
Algorithm Hash digest
SHA256 e388e7160075376789542e7b23a08d1a0292f34532fb582f3c610b4ccf80d8f0
MD5 f78000e166a20f7c4044ff93e23f489c
BLAKE2b-256 1c3de945e42d42623dc167c9cef6392fea6a4ce483aee9ee0f26e917426a9efa

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.6-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.6-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 cf4bb1b89a73b1fbaf65d0606f2e47be04c7e32eb3f5d6038c00d3032c4a2de3
MD5 69da01be5ec894e3ff96a4157c443746
BLAKE2b-256 e3fb03891d997e080d5be2330c91c8c47be60bf859b1ddf71ac818d9e431dcfa

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8c75f8acf6a4ea6034d642d5235cc99fa60bd5ea7117a0fef444c4a915ecfe7a
MD5 44a8f583dc8a3c6623af431a24c61691
BLAKE2b-256 e3786606511b4a580277185dd4cd5d052f63fd98c25ad0f7ab414705c28650d6

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a71163a61350c75b7543c0a4ed84b5d506f1df03f75fac1a8f74b13161698f65
MD5 d2b30b168c0c53209c99c88bd2dce9a2
BLAKE2b-256 af3108853865a27a0b2ee46cf1ec2c6eb89043a06f17c5b7dcd9c3d00c847d19

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6c611cfdf0fb38cacaaf21cb7f207d7f01fa806f8ae822c923afbb49897d4124
MD5 9edf245362974fc0470ac4e9391cde91
BLAKE2b-256 2bac3e24f39e7714dbcdc0e7d2a4e5573ce0962c2af66a584cd69417bd729394

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 3c45291007cacbcb5c46ba22476e85fe6b64efbb2e796079bd619e891e00fff1
MD5 f53523a5d5608e17d085bb9978994ea1
BLAKE2b-256 2159dfdbea93bebc1dc52fa234774423d9cf83d30ce306d7afe6f08613153fa3

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.6-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.6-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 cb8714ac35f7a52860dde1c767930e568d4a8f6faf503763a06970ff59853259
MD5 1ccdbd840335715465a7d23c803e5fa1
BLAKE2b-256 4c672303069e8e93978e0b9c6c6381a53d33d61b4cdfde577e6e19829112bdd7

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7c22009aab27f5e78c39b2281a7a6add3a26dd02fb71e183d7f1a2997a7ff458
MD5 ae269e7d460e0fe446e5393c2bdd0697
BLAKE2b-256 93e4e741888c629c48688825bd13c7c952cb3039ab18d6c01686b19103551bcb

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a85bcb247fba4d833aeb2a82b5fe47f4c5545c2cb61dc37618d0739b4563cd56
MD5 49d1019f5baeadb570dbba579d8e9e56
BLAKE2b-256 0234948498cafb3c521fc21cbb8a6a98f82dc6bef5edb2d14cf52af47bc2df0b

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e3a6bbdcddb99506de0df08ea21b0e98202534f08454d645a644719d6b283a2a
MD5 dcde4e6ea807c5f1df5fa6bce95b74a1
BLAKE2b-256 570d9d1799da7456616961f80a135f6656bc515180a3d303bfb44dc4ad15c782

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 2855a6c95d4ca919e80ced065b8c193569d43ab4e199a662a4cc1e213492b1d6
MD5 d44380de324ed547f8133b140c6671dd
BLAKE2b-256 2ae8bbd6b1cbd3fbc50f6b5d6e3c52b3fceb6ce9a20d7360bbcf8247e91abf77

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.6-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.6-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 74425f674446304270b38e326a1ce72d2ae0e80f142dacd46ce2872a99fad671
MD5 a42a600f6d52a3664155f4fb4352e149
BLAKE2b-256 e6f2c619ac4c85ed6c84976295ba9519c248f8f66d7b52e815bf90826a10b772

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 95ce6b09399c15138e19e00689a65ba2b9e96a92212d100b7bd20ba04658f2d8
MD5 a057148c1a251ec15b1f129b0056ce16
BLAKE2b-256 b72947e22c89a4a005f19261975c45eec8793cf0b2c514bcab2f445b1868d019

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b107ea9ed29e7a66908264fd6501371b592afb7dd381f51165cb9bedd932d435
MD5 9cc3bc616d49ae421b09f2f7c9897dfa
BLAKE2b-256 a1c6e5653c9fde4bb53d286dc2d523acb30a33ecd3435090090f7083276f194b

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 690d6f8bd31270aad7cf019a6296bd8da773f0bf3c7b783e443f7177aef74d98
MD5 30b25f91a784ac3868b57ab2f77d57e4
BLAKE2b-256 4941a570926399a519e668b9df0704340b1b6598fc3547daee56922430e0cd03

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 cebcdf8388dd3173edf0a59b452402ee1c1102fc21b03215eb926ca02e89ef87
MD5 167ba4d8590ee71464959c1e7074a0f9
BLAKE2b-256 ac3e47bbde8b023be5ab5c50a18e71b987e3a8d98d70ad2915cac0073193af92

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.6-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.6-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 b416be54f5770fdf812b463617134a4637ed3d995659b86970a91228150692d1
MD5 400042bf67881b7c94ba4329d3f204bb
BLAKE2b-256 67c8ffb061c87697acba4413907e7f7f24026fa6fa50d37d1ff1189e5aae25d1

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0f855dc9ada046170f96ddaf63d13ad67f5f97f93c78e4363dd15f6ca25b1f30
MD5 a6b60539fa83c805fa69d064e13c09c9
BLAKE2b-256 235fc8df9f0d3873dd294c66ef4aff371c997bd26be4e31236e3f837358ee72f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2e1b00222457516b2ba9a7d205e7be2680fd10caf5242a48fe701ba9beb851d0
MD5 2766f602d8bdcf06ef67240015044e56
BLAKE2b-256 275766a3b47e692365f99bc4cd11997f741115a3585ec46f4f96c3418485efa8

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b6d006b53872d25de4f2a84ddec6f0ad9df7ece77f130cd724e32c9913dfed28
MD5 0454c5a5846718ad02b0a1f0ce503285
BLAKE2b-256 9c4f1cd889475810d82033a620683478e5e3a091a2c4887bede125bafe42b1c0

See more details on using hashes here.

File details

Details for the file blosc2-2.2.6-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.6-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 391322885b9b27e20372b955dc99a443acc0a585857ea6ecf00909b8a54325e3
MD5 afa4eac9d72175828fda1796736be904
BLAKE2b-256 f1d4b79081b639c81472c6a3cbd551cd3d906c16fa625278205015aef43460ad

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page