Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.6.2.tar.gz (5.1 MB view details)

Uploaded Source

Built Distributions

blosc2-2.6.2-cp312-cp312-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.12 Windows x86-64

blosc2-2.6.2-cp312-cp312-win32.whl (1.9 MB view details)

Uploaded CPython 3.12 Windows x86

blosc2-2.6.2-cp312-cp312-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

blosc2-2.6.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

blosc2-2.6.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

blosc2-2.6.2-cp312-cp312-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

blosc2-2.6.2-cp312-cp312-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 macOS 10.9+ x86-64

blosc2-2.6.2-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.6.2-cp311-cp311-win32.whl (2.0 MB view details)

Uploaded CPython 3.11 Windows x86

blosc2-2.6.2-cp311-cp311-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

blosc2-2.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.6.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.6.2-cp311-cp311-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.6.2-cp311-cp311-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.6.2-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.6.2-cp310-cp310-win32.whl (2.0 MB view details)

Uploaded CPython 3.10 Windows x86

blosc2-2.6.2-cp310-cp310-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

blosc2-2.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.6.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.6.2-cp310-cp310-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.6.2-cp310-cp310-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.6.2.tar.gz.

File metadata

  • Download URL: blosc2-2.6.2.tar.gz
  • Upload date:
  • Size: 5.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2.tar.gz
Algorithm Hash digest
SHA256 8ca29d9aa988b85318bd8a9b707a7a06c8d6604ae1304cae059170437ae4f53a
MD5 dc4d59f81e26da1d8e2aa2d204bf3422
BLAKE2b-256 35c44f634cd28ed043be0721d71ab82b8fab572cea3721f0ddcf365e1a96aa3e

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.2-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 b99157758b5d3ba11c46db26602750555053aee2b917ba3209eaf37ee266ccb4
MD5 90d10b2460aa24e7b4fda530a868b013
BLAKE2b-256 d93eb4164fc2f68ecaf766133a9e9f0b32ffee983f1fe9dd540fba1e23737237

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-win32.whl.

File metadata

  • Download URL: blosc2-2.6.2-cp312-cp312-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.12, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-win32.whl
Algorithm Hash digest
SHA256 3025e4d0bdab498853e0cf971ece10ac5709c875f0b6b4272fe069326b69ef42
MD5 aa31c9dd629c919140e10a63ac654e2c
BLAKE2b-256 0aaa4c07aea2c2735dfbe2098c6f3de4adafca096383166daf1eaebaafb122a9

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 8a63ad8ae52af974d4ffa1490aa7715cfe0d6408363686fb141ff6f7513bb0ad
MD5 38ce56883e308d4ee568160b288f26cd
BLAKE2b-256 fad998aaf7c9445580ad8bc936739700c0619cd5401edb691604aeb7745109fa

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9c4fec36f58267fa0b5b1ed7f688469313e5af83ed1cc70ba01001d3fe4b824f
MD5 f6795fd354fea99915b9aac303b23f3e
BLAKE2b-256 f8786a5722135283264d5a52fc08f9f47bef96cc7478229d43fbd42ba77edb14

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 6ca4c24cb1f64dba1b900fbfc165649bbfd9c890d76e356a682a9cff4c34f967
MD5 66ba09cecdb8451962188a0b4c75eee3
BLAKE2b-256 ae5101dbcb0b0af327b6a8eeb05f3d3e335233ba1d38d2c4ebb3f04b73b27705

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f4c2272915e0f28cd10258393506cc31616317d94fed77b61617c98734588016
MD5 b7d2bcd999982a901b8bf92d4223340e
BLAKE2b-256 7c193de7d29076d3eedbd5060ed9bd551fa488dda438090fa30b7a671b15bca2

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp312-cp312-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 bb8fc5c0420eab9c4c0c7eddf1b8747b817f7aae5145e3e99607918af3f42588
MD5 84e3ff8b855b3f55cb47aa1ec7a16fcd
BLAKE2b-256 64986f0e0e66f395c5123c74c7acc4626622e34fa1ddf2b45f0684d7a9c81bee

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 db38cc7aed6547f0855ef5dbb13853f653a91174bf5e79841dd00ff1914a83d3
MD5 88e355762d7b84df834511767749e2e3
BLAKE2b-256 916ce43be26626f5c2b3eaff83c2b652316e0884c9733191df74aac812ef91ea

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-win32.whl.

File metadata

  • Download URL: blosc2-2.6.2-cp311-cp311-win32.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.11, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-win32.whl
Algorithm Hash digest
SHA256 6c5b861a8c51af1cd7eabf59c3bdd944f873ea5de8497602af9c5617cabe4f7e
MD5 6ce611e69b5b9cd4c9085a02ec85c66e
BLAKE2b-256 0af42e3f0230eb961bd88c7f52bd091627f3c4bc3c590af78aa77c176cf84f54

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 be1925fbf1ce37d384f47d3f02710abe79cc7722d09c2d50044845947e85d2fa
MD5 ce9a9abe0c595e45629ba189bc96ffc6
BLAKE2b-256 9a820634ae77bdfa53419b8c0a56dc0546f8eb8d5b14e34aab3f2600fd67de8f

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 81b0d7cf029b097fd130817ddae66e67a92253136812a5dddba3d9504bce15ed
MD5 893d930b61f6eedee0e40a4d14275228
BLAKE2b-256 31ff1de7cc9606793bc85d234b46612925246927909adda96bd85bb42c39fbdb

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 50661d5e9147b8f50a86c7d86ec2be907ac33418c5ec82963f4487d851e9c88c
MD5 8bb5b643ba44715a9a820b255206d274
BLAKE2b-256 7747dcca6bbe796f386fbbd16520c57feccfa672ea6439e4848655138ff69610

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 cbe8e97f0bc94a45456f186c374e5fb91d35123ebe80e530d849d1da95cf6770
MD5 0f22a240a739013deabb25f47c7ba20a
BLAKE2b-256 2440f81eb3a8bbcc95aa7ff82df21f9d90a6819ee718fcecdf2ac6335ddbd401

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 15d91ba9fd24391a67dcb1051b82490b0cbde3a1d473209fa578e7a96d801bf7
MD5 6eab8a79d1aa94f7bdbc95b20462ae34
BLAKE2b-256 b8c01b211ee0cdf2b57f6edf8ad5edd5e198d215d4828fe8a8cb7534c89fa425

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.2-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 8291cd97f3730873c498df610acb0177ff11901e09771197e1eace5c3e1b9669
MD5 b86a50ade36cf4aa0a7a49ee04488152
BLAKE2b-256 207326ac0a44dc7668c5862728a46cf2b9b51ab69cd1716ca348f4a11f79d325

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-win32.whl.

File metadata

  • Download URL: blosc2-2.6.2-cp310-cp310-win32.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-win32.whl
Algorithm Hash digest
SHA256 abc87b8bda70290a33b0d5631121d189f90046b86f7992865428672471cccba0
MD5 71cd6694690290e76a1fe8249d8dab39
BLAKE2b-256 0b7b353fdacd5efcc96e9bf4b7374e24a050b2ab6cfd77439060c3d020c67f7f

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 afb962aef4f2b3b5cd20a3ae2d92311bb829836ae283b5ac595fa14dd2fad47c
MD5 5fbec00f6e6b726e096732f05d8931d8
BLAKE2b-256 7d8b1de981ee44406387b49e3609588d5743913ad32ca1cfa35a8f7fe27d5551

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 072a8753d627499893129d480042a61ee47845ce99106fa0e7d8ea4f0ced37a1
MD5 41563ca023ef9422bbf72a932a9b42ff
BLAKE2b-256 46836daa9f0d8c5164c01a61f736175fd79def0141aace3c12446ec0f9ebf4c4

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8aeb8eb12c60522bf0eb6d49687aba925e710ba4f9976cdde519d7af3bc547df
MD5 40b4d728c49d2e14d6109efade2de665
BLAKE2b-256 660b6593d4c6cc3be08fb02bfa540197300db822ee8cf19b393d3e1e17bd75b3

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 407627050d116d1cce85b197616350d3f2852f7e036a4f59a97d5cc07f345ead
MD5 3b539dad31afb65646f93ca5aa25b131
BLAKE2b-256 4c80a13e60558100f0617b5c14ac86222b1f162be7ad3b9b9fa67be4ec792a1c

See more details on using hashes here.

File details

Details for the file blosc2-2.6.2-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 00db67006601f534553a7948213595f384eac0e3afa41a4f5600fbb3ba580ae2
MD5 6fccc467605f000819b60d9eab4ff33f
BLAKE2b-256 c8a76056dd3c5a22dc1cc4e4eb95203399215c4c9d27e29716e86696305ed631

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page