Skip to main content

Python wrapper for the C-Blosc2 library.

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.


Enjoy data!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.0.tar.gz (4.1 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.0-cp311-cp311-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.0-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.0-cp311-cp311-macosx_10_9_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.0-cp310-cp310-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.0-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.0-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.0-cp39-cp39-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.0-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.0-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.0-cp38-cp38-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.0-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.0-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.0.tar.gz.

File metadata

  • Download URL: blosc2-2.2.0.tar.gz
  • Upload date:
  • Size: 4.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.2

File hashes

Hashes for blosc2-2.2.0.tar.gz
Algorithm Hash digest
SHA256 4cf3893fe85da4bd899ecf7a457fd09b2212fad256182100108de7a92b88aa87
MD5 1c3472b35c5c326233623c3a8236fb9f
BLAKE2b-256 b6205141526d669b68e7530f417f55771e4bb0bd81fe6e78ccd851d6bae353ba

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.0-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.2

File hashes

Hashes for blosc2-2.2.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 5fb7f8075d2ece5cb64ec1a5646dc0cdedfbabb2ea9c03ba1fe88fa58794cef2
MD5 628ddcd16bd278cccfe64a63899db912
BLAKE2b-256 e09edefc6e0d76383d84a5b113007531668feb237303cb1270ea25c5b368f6a8

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 79b40ff98a5c3ce88aabf5b18a76c5ef88b77ce1e6436dc0c0b414b1bc279487
MD5 64e2055405ef073b10fb86e56d7abd6f
BLAKE2b-256 f6190151f805ac394119cc281f0fb05ac59b1bb971743c5c0c64df48d2823cb9

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 52ccee1e3d0337502f6df14cad28a9f747f23b6ff21676ffc77c936714264b87
MD5 620eeaac924aadf88a18e9b3166b1d5c
BLAKE2b-256 654984fa4c9636bf9ee313f6bedda95b1a82f0885254339f915afd2264917e2c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 195d56b75b818c19f196b3e22225b96ed681fa8dda6a740b1966712392bcc445
MD5 0ad26ff04068bc6a294cf189b680f10d
BLAKE2b-256 2e9fc9c58ad786ce128e4fbaaca7e648a218c186ddcf3967338cfd4c018af854

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 67b0ab262c5edfada2ea82aeea5958e7ea06eec0cb3a122df3ffdae8dc6de116
MD5 2fb65afc5f9ac638da8c140cb565f713
BLAKE2b-256 3ce5f80108dbe4b5185114eb64f6fb63f61ce6d03ba9ea90c30a40d7d855c92e

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.0-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.2

File hashes

Hashes for blosc2-2.2.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 404d31a4d057b27463b804a9a2975f67579258f849e094b6e92089d8a67c2a62
MD5 b3db972786438d0e4321fb275f743619
BLAKE2b-256 6dc0e4e483b6cc98ecdbb2d7f0829c0cb8f187477a371845897654e70d700dae

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e57ff284b937572f1d01e55ac18bde9d81fdf488c38bfc7cebf9e96454c52e04
MD5 ed234fc3bbeb9384853635a232da26d1
BLAKE2b-256 7619e702454e4b1e68987d61948ea4f05944bf153c09fafb48120ca25df0b763

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 6758c4f812b7934b071653daa40482461550cf32e06e88098c7fe4b0d11f1247
MD5 51aa61b0e2d91a3c48de220eed07fe79
BLAKE2b-256 c7caefa03f104c44c04c0d5d81aac68e5ef7f832705cfb8d5678556f1e677707

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8b3396334779ebb41dd8722ed165bb3be5da4b6f09420a01bd4d2b15d54bc4cb
MD5 c2c35b2ee1b075d001ea42945f70cd8e
BLAKE2b-256 60c77d92041281f808c0f8daa6cbce86edcbe08aeba223874fcf539ea0992f39

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 317608ae618101899274bd5b4d33c5a8ff3234c9e09da66549e7c67463b41fca
MD5 efcb98dd389e92e5ef7d1827deccdaf6
BLAKE2b-256 dffae6766d47fe578d65cc2b528252113e35f1165982e768bec331b729f9f046

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.2

File hashes

Hashes for blosc2-2.2.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 e85a7084f802e04e7fd651975aef8c49ed5d39f58f83ab53b936c99aa7d431f4
MD5 80d033cfec6d512b34fe6eb26d48ddf1
BLAKE2b-256 89073fc069a66462f4bd8e46a7c7908a091d777e85b81d4a6a66da1e4649c8f0

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f49007f69c7ae7388cde9ca11d988b08e669493665db2b654773d79af58da0ad
MD5 550a3ac4547014091850dbc307913127
BLAKE2b-256 f25ab3f2fef4a4188bace3b26d1e2b2679c6815551223ccd4e222e8f6c53ef88

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2eb94c50b6f507523f03ecb21f9999472471ded04db5200689d06db191c86686
MD5 54f0d0c201e2125b335f005f9fe258a7
BLAKE2b-256 8e689d21d29e504cd36a1ad9669b91e8b7043441a6b2c489b5a0300db405adc6

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9d4a068967e443c3999652f0b946b0c45e617772ccf84fb43fac854919bb72c6
MD5 9bb2cd078a0c798a7683f979ffc829bb
BLAKE2b-256 2bd786e91774234918378adece2794f04b21c2efc216a6fb4c6d32403012dde6

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 4a52b9f26f00f47b4219192c9192741f068da59db62fd2e7e237434fecdc0e01
MD5 efce456659f96625d1443edc2d6dcf76
BLAKE2b-256 242c845d0e3fb16f69045a8aa6b030226d059064f94cde9b1c0246a22ba89c71

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.2

File hashes

Hashes for blosc2-2.2.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 094c0dd5bb7ad47f2b526d2b43b36768db72a46b3de0a35953d80e40091e51d3
MD5 f648a1c9fcd1377468fe3481d43999ac
BLAKE2b-256 db0259e5b956c726805669323de707e12c703b892504acd5f16f897ad4c85fcc

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fa7d90390ffb2d4680c9c4c079316e68a8f3177d6938b35da3eae733f0132ef1
MD5 2c81db55cc7ba42a85e13fbee81eb3e0
BLAKE2b-256 4a8bbf28f7701443ade7029407cc19460ee585e1099bbadf494f453a28670003

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 436dad718506a7d9ddf42cdfff7416aef8e2dc97ce65c1e4f221911449dcfa5f
MD5 3805c81e34f6d35bd1b70f1386869d17
BLAKE2b-256 c8a5b1885514fd7b56aea7c6e6009acfffe9d9f6be8518101e69f0e2c55fa8ed

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 510a07cb62ea9dca928b366f346be3849d729fa438c3d500796acb100b90f042
MD5 1f13e5e0a4d77949e737029460bf0b42
BLAKE2b-256 6dd7031a7b702ccebe14857ba54bd38f9dd8c8e9ec69a2e73e619b8b71e214e8

See more details on using hashes here.

File details

Details for the file blosc2-2.2.0-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 aac779857e2c6715ba74ec6e72687afc387f9a5dbff90c4ae3c8a5b7d3e59a6c
MD5 66765c4b12546c807e2b1c02cbc42848
BLAKE2b-256 9770162f1986a5b79067cc5c553906b67fa890ea21137eb3af32903e9901ad18

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page