Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.6.1.tar.gz (5.1 MB view details)

Uploaded Source

Built Distributions

blosc2-2.6.1-cp312-cp312-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.12 Windows x86-64

blosc2-2.6.1-cp312-cp312-win32.whl (1.9 MB view details)

Uploaded CPython 3.12 Windows x86

blosc2-2.6.1-cp312-cp312-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

blosc2-2.6.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

blosc2-2.6.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

blosc2-2.6.1-cp312-cp312-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

blosc2-2.6.1-cp312-cp312-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 macOS 10.9+ x86-64

blosc2-2.6.1-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.6.1-cp311-cp311-win32.whl (1.9 MB view details)

Uploaded CPython 3.11 Windows x86

blosc2-2.6.1-cp311-cp311-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

blosc2-2.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.6.1-cp311-cp311-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.6.1-cp311-cp311-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.6.1-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.6.1-cp310-cp310-win32.whl (1.9 MB view details)

Uploaded CPython 3.10 Windows x86

blosc2-2.6.1-cp310-cp310-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

blosc2-2.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.6.1-cp310-cp310-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.6.1-cp310-cp310-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.6.1.tar.gz.

File metadata

  • Download URL: blosc2-2.6.1.tar.gz
  • Upload date:
  • Size: 5.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1.tar.gz
Algorithm Hash digest
SHA256 eee6e321ace1f41db3d3290f92d97d86c235f2c43a873e83f9ad0281b7f1c007
MD5 740981805daf9297cbcd1535f511bd41
BLAKE2b-256 87993d9e1b25048a89acf7837267b787dde5883b22072be944f5195f0af960b3

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.1-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 cabf6d9d96f25741d78348d1485dbdae963f98a464e953fbf645a103f7742e8a
MD5 8ff1f50d911d74f39af390886d048a03
BLAKE2b-256 4c6c3002c279a7f00aa9fe60da0375fe337b56f4287855db9f63ad0f2678b9db

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-win32.whl.

File metadata

  • Download URL: blosc2-2.6.1-cp312-cp312-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.12, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-win32.whl
Algorithm Hash digest
SHA256 f7d0b0db44d54ee90889d4f9b2e76892ad0437d87bf01602d254c8912886e47a
MD5 377f29bd69a3767a9081168cb410e5c2
BLAKE2b-256 ad03e35bb80078682088790b67ba47d377d4e67da0c2a25d6102a5f0b48a9de8

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 28286e8977e5b92068df181db69546e4a31269b01794df1d3f6b9ee592bf0c2f
MD5 ecc8f9c11a9af31764d7b4fe16d105fc
BLAKE2b-256 e21a381729dddd4967c9fc5ba841adc8f53471be64f09ee33eb075bf52de7fbe

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 decdaba2ef3fbfcf6d1c276c8c9de37bbe0275daefc3bb27a5234800bcb77c8d
MD5 b19c0602e18b354b953f5f11ba2dd376
BLAKE2b-256 8687905eaf7ba97d8a53f61005d4a03ccb0708bd4409e1db44a18421679ecbdc

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3f251ffcf12db2f27428512223c63a40e881a659c72d729bd22ad2ce2f4f38df
MD5 e4aa98d142cdd07b31297f59f3683656
BLAKE2b-256 6a2e775c3cfc705f419f8fcbaaf3ab49661a535e320f00dbb609dff14a4b12e5

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 09f31a31237dfc58fb6f05d61dcf5f8ead5bd84cb3ada7a2b64adab62b4ce53a
MD5 bb3f853ba395f31be99111a22162d26d
BLAKE2b-256 13f11c01ee8b3a49e76c895632fdb2487ca57c274c304afb5832c59c15c765c5

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp312-cp312-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 1e2bf289a867b346d240838eb4a8bd83de759be2b719c5ec581b7e4bcbb6c99d
MD5 d81e107769bbac85f7611e1e92308a39
BLAKE2b-256 3fccce7fb1ade08c1244e2de316de8ba6cc5637c4359b1f7a389fc2d7acc8d3f

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.1-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 c23087b13693cb6a55437dfb8c025752f5eb0a1d63a65f4a15d5e8bb416d303b
MD5 abce7fe34da7c505694603aedf0872e6
BLAKE2b-256 8af7f65821842430e8cae728bbe56563463f1e408a229ad31878b1047ab0625c

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-win32.whl.

File metadata

  • Download URL: blosc2-2.6.1-cp311-cp311-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.11, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-win32.whl
Algorithm Hash digest
SHA256 30eb55008b45d1e4047143fd0f4e1866993d59c03eb11c9966ec25ca3973bd02
MD5 f64d7b4fb2d23ce0f10b0a90e872e300
BLAKE2b-256 566f85696e30fa93ba7a0ccd396023f4930ec79d7d1a59e39b18392311bbed4b

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 df38c013b26cf0f6c5a6cb344420bb496ba0ce99f2fe3fac5aecfc9fc5969d91
MD5 e4febfb5c0b92d59738d86bf0d91d3df
BLAKE2b-256 1a9194c4efa85e9564b6d86dfec3a65daf56f7ef3847ecf6ba16270f7ffd1576

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 95d8dcb1c6e565ebda1119e60c4a87a3cd6a49c821bef519fb88581eecb12a15
MD5 c64ddaad86fc98fc12ef5887eb05ac17
BLAKE2b-256 6bb52c15478d691081aabded334a765b92abdb806a8d994250ea236e49b7b377

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8e06eea3f81ef9ac15490b97f8153601c7baf9bdc6b31968f2312650ec5b2e31
MD5 02c8954a64c7eabe74a862ccdc62b4a4
BLAKE2b-256 bbbb6f1fa41bdb5a6a821b9b3ad28e01a4c6713626696f6ae727f0146b72d824

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 d1f55567884b6c24da1c39985109266253579e432c2530994765e9e6e1fd9011
MD5 4d3ed02550df83f1ec2a0c5f50e0bcb0
BLAKE2b-256 dcae124b3d3878ac62e60cc97c150c0d4c6f4867caa42170ed1e602ede132841

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 5de94c6d05caf4df69e215d4522d1645aacecad05deeb34293f21fb0017f3454
MD5 6f9a28d0c194766cbbdc1f1cb7f9e791
BLAKE2b-256 2c51d1505843689b2bd956e30a0556c7f716c4d0e822691a86d5a5f22b42acdd

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 0bbd23c793c5ceee5b0ce4a2f77e0cf7fc095bc1b2071ce271278524f96ceee7
MD5 6c1ef27552260761911fdbd62d4eaa4e
BLAKE2b-256 de9eed12a8c33599148370c68bdead54f318e8f763fcc7dd6296e301bdc782cf

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-win32.whl.

File metadata

  • Download URL: blosc2-2.6.1-cp310-cp310-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-win32.whl
Algorithm Hash digest
SHA256 38552c52cd043962e6d2990dc7a398d4f946c1cb9c009e5cb0b4a9601e896e95
MD5 0800a03312ca44578fb31c3cd9a4c558
BLAKE2b-256 8e47d1da96f8359405c264b0b38f9eae18c6f87ae10d342df5ae4b13e97fe201

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 a6959271f336059ad6d3164e0c6643ba05c4753d42a119c42ddb68d06c26a3e7
MD5 df30fede055f8ab88fad2f4b4edb0c29
BLAKE2b-256 3a02457b1974b87d9d07a506ffbb6033e31ae9721d4e5edeb5fc99f7f1d7fdbf

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7a476534baafe332db4997bcf53b5c83716d857af05ab1abace24a5c69c968b6
MD5 5f5620ad1bd9c17819c7a832c639de5f
BLAKE2b-256 c896c33195ceab967a295673101f0ef06faf78aadfd5830ac7557906d13a0132

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ac1af499394ce852d66e9f819d8333fbe52e4bb7a6010d744cf68b912fc9e91c
MD5 e449fce13a5a0e15b0a01ea9ef9498b9
BLAKE2b-256 30ba01a8f99834ca3b1cd79c9a0430565652f8c944bbae8e3fba770a141bf802

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ebb85bd1b0864d5abe53fef3319306b9a89231d5295a31c50d845bba37c985c1
MD5 64c9cc7290dde84d5c9d0b6369a92bf5
BLAKE2b-256 34c226a9e0e8856dccaaa935b8cf5641a2b8e5a2b56c490cc8e040fe1c1b8738

See more details on using hashes here.

File details

Details for the file blosc2-2.6.1-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 a4be4398b2f39a336958e1e1ce8aa59923d1137ebdd46dd7821390e725168300
MD5 0035e2c61e6c2d668b46687c7d2579e6
BLAKE2b-256 aa4677a86f29be79f56e098f4dcb2d57fe9fc7048445b4c59c7e91a720393138

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page