Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.7.0.tar.gz (5.1 MB view details)

Uploaded Source

Built Distributions

blosc2-2.7.0-cp312-cp312-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.12 Windows x86-64

blosc2-2.7.0-cp312-cp312-win32.whl (1.7 MB view details)

Uploaded CPython 3.12 Windows x86

blosc2-2.7.0-cp312-cp312-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.12 musllinux: musl 1.2+ x86-64

blosc2-2.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

blosc2-2.7.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

blosc2-2.7.0-cp312-cp312-macosx_11_0_arm64.whl (3.3 MB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

blosc2-2.7.0-cp312-cp312-macosx_10_9_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.12 macOS 10.9+ x86-64

blosc2-2.7.0-cp311-cp311-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.7.0-cp311-cp311-win32.whl (1.7 MB view details)

Uploaded CPython 3.11 Windows x86

blosc2-2.7.0-cp311-cp311-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.2+ x86-64

blosc2-2.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.7.0-cp311-cp311-macosx_11_0_arm64.whl (3.3 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.7.0-cp311-cp311-macosx_10_9_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.7.0-cp310-cp310-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.7.0-cp310-cp310-win32.whl (1.7 MB view details)

Uploaded CPython 3.10 Windows x86

blosc2-2.7.0-cp310-cp310-musllinux_1_2_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.2+ x86-64

blosc2-2.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.7.0-cp310-cp310-macosx_11_0_arm64.whl (3.3 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.7.0-cp310-cp310-macosx_10_9_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.7.0.tar.gz.

File metadata

  • Download URL: blosc2-2.7.0.tar.gz
  • Upload date:
  • Size: 5.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0.tar.gz
Algorithm Hash digest
SHA256 9b982c1d40560eefb4a01d67c57e786d39a5ee9696f3deadd32ebf5f8885eb2a
MD5 242f9b65454188d9d8d0693e6fe20001
BLAKE2b-256 b9a81aabd9a9bb8f5f1be24e1a4377998eb9457771173c83b39fd19c6c19b602

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.7.0-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 62d2a6eaf1be1858993a4d7b2b8efd2ede5c4eaabe030c611cd075d907aa5400
MD5 6ceeb562c0779e9aa5b6ec9b1622add1
BLAKE2b-256 60bf6ceba93b39a716bca4feabccb034292f69fc06fb02bb86ecdb828cb95881

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-win32.whl.

File metadata

  • Download URL: blosc2-2.7.0-cp312-cp312-win32.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.12, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-win32.whl
Algorithm Hash digest
SHA256 407896867032a760dcce6c25d5e5a56b6fe5235245e065e2549697f69b5117c6
MD5 512d1181e8051402746dc0917c309fa6
BLAKE2b-256 3e6be23649ecfe45d58558f11d67abff6ce390ecdf612f5b1f698a49d44127ab

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 1546c04d25ce793fa0fd7a83999bbb576ff84ef474fb45801f0b6dd76b84803c
MD5 aedf4bf6f582cd43cf67b4d9bbd24ed0
BLAKE2b-256 b945935e902158ab55df8d4c764ded202a9f32dfdbe683e827cb17ec2379a78c

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c6a700f9324b37e814c5633c43b081c60962f4dd59c0340cefe5f61f9f0411fd
MD5 30de13d2ecad9dc9a54cb20c9744b588
BLAKE2b-256 95038e7aaf9295c490b495ef9663d4e6a53b7b9b669c856425275a8a2ed6dc10

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 770733ce68d82674d1f80961fe56f3c2d914d8ea4de036af3888a22479add97d
MD5 89aad828349a29e26861d8ef3762bd1d
BLAKE2b-256 f94159e615cc6fd04a3c7a9c28cf082eadde713fb50c286f4c89335981b0e4fa

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 64a26c9f7a4a5ddc5721a75b37f913f9e21c0dab96d8c152a64f8faf8659e9ee
MD5 178f179ccb1d4e317bf021877feaf061
BLAKE2b-256 f65f3bfc80bf16302a290324bd3e44512702a38e60a9d0d1cc26de0f18e46fe2

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp312-cp312-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 729305b06e76b0c95b0ea5090aa7ec87eff72ca43e194283e0cccee92bbdd1e6
MD5 d05bcfd40e77e806453ec99e62534b97
BLAKE2b-256 09d1a17c42d6d2590bdbe456b77b3d087e7263c218c1737273bb47cc3eddbbe5

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.7.0-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 05d40ede9cf0ecb25500cfe9bebe190e75f246eb1fcd7bd358ac1acfef44ee7a
MD5 bf41b6858b0bb309ac2648b78a88d0e4
BLAKE2b-256 0d4ab013e712956794c8d175b10dd89a932d11b6901bbd26a290577193fef2e2

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-win32.whl.

File metadata

  • Download URL: blosc2-2.7.0-cp311-cp311-win32.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.11, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-win32.whl
Algorithm Hash digest
SHA256 4518944374880d822f9ca90d4473bfa9f4d884b462f78365e224c2b291962e44
MD5 2adab866db725687ec96d7308eefad56
BLAKE2b-256 330c204d65873250e4409fd3e5611e54d115e301fd7ea2929d72006d4a290b2f

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 fd3ca9a61bce4e4dc8006b613fa9dd8982f71e01fa9f593d6cc44d9fdbb56174
MD5 488b7f6518f9529b2073e6661e748823
BLAKE2b-256 a94a8b9db123d7a514eccd9c366aa9f14103b726226ee4e1eca33a4b8d8af459

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3d4b208d5f5947d3062d3353717c43e0ea8e6ccdecdcd30737d5305628e0062b
MD5 0022ccb4804cf1c696c9093f241af266
BLAKE2b-256 365de74303d76a81b593084dbb6b2767375ecc1e007d7e877f4bf4552662d8aa

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 634bc22f17ae47a166b8201c77ba11bc160d9997ace51fc820cb3cbd285d47f8
MD5 5cb5d2de86774c19fb1748305287f0fd
BLAKE2b-256 b8cd1a8aab659f7688c277a2f3655414d7a5564ce701bcbb0db4dfdcf09e543c

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5b640fe2d1d39af2dccffe5e100ef94d21940bfb7f0af44ba17fef718671b267
MD5 5862616d5a459f5f7c1b060aa5a76878
BLAKE2b-256 8a2418d1925f9518db6beb77b26073712b208a318931e9bfe7c142c54afdd1f2

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 565701ad336946a7ef12250def97aae2257de1da34ac8cd570be91b664a03d30
MD5 40da42487a7f159e1e45b32d17340b96
BLAKE2b-256 eea727ae33a06b4d83838a6675b433f3fed4cbe13dea6488bafa6af235a435b8

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.7.0-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 17dd39f62f1686a170232ac8bcba40358ef67e919a91fe840ac71a45d067df30
MD5 b51e66821fd6737d35df125e2cafc9fa
BLAKE2b-256 e4f13e637d84af04ba6feb81c0cdceeafa91f3493a886c28260f3d9ae8dc617f

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-win32.whl.

File metadata

  • Download URL: blosc2-2.7.0-cp310-cp310-win32.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-win32.whl
Algorithm Hash digest
SHA256 488dc4be3b6894967a7189952634644f8da46c4bab7734719d379cdf5b440dc0
MD5 1b752f0d724d3cb748c5c77e34366806
BLAKE2b-256 8dcc7a72841f5ed357f6dc0af208879658bf22d58148d05f2bf0e67f65773bb9

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 a9518b7bbaa0f9903a5a921abe6abb0faa56b0e0ad2da0416ff3a486a4b2e0aa
MD5 6f5a89703c042c662893d2769c24543d
BLAKE2b-256 1be21ce65e3c7295c1fe3edd98caff58f4aff27d0d6f9ba06df233d2df13c216

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 928a89851b8528ce9c233048d832be5b2fef47645d5a389c021f3f58333fa3f8
MD5 a08d64ef33665fd5363b44ab8a7083d4
BLAKE2b-256 44708df5f8a79c7b9019654e3d890d4ffcab740087ccb88c5df05f7f79ff98b6

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b1ac66ce25214b0b2e53beda9bc6f333dba16f2667649b1026ae041511b5a07d
MD5 22b7165ff1e99a7f78ba746df2e4cfa4
BLAKE2b-256 cb250adeedb1ae43f40d4e4d1b093dc1606a79b5abe4a49ca671390aa1c7b06d

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 18e3c4c95fe40ea9cda88c784d96e4efc8ddf53f94074cf46daa2e91c9ae5137
MD5 8827596054b33db1d02d76422eb79a67
BLAKE2b-256 d534c14ff112a6a7ac6586136e6dc07ba0b9e29b144b9a9cbf7b5db48f894364

See more details on using hashes here.

File details

Details for the file blosc2-2.7.0-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.7.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 aa71042277956199676169335eb64aa76e33adac5a22289eccdb7d10edf402b6
MD5 2b6fa87d84a447a8d69e2ca364954495
BLAKE2b-256 b1555577554cda74e7c0508035e3b15a8cf514d731548f67c526d18d1e2256b2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page