Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.6.0.tar.gz (5.1 MB view details)

Uploaded Source

Built Distributions

blosc2-2.6.0-cp312-cp312-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.12 Windows x86-64

blosc2-2.6.0-cp312-cp312-win32.whl (1.9 MB view details)

Uploaded CPython 3.12 Windows x86

blosc2-2.6.0-cp312-cp312-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

blosc2-2.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

blosc2-2.6.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

blosc2-2.6.0-cp312-cp312-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

blosc2-2.6.0-cp312-cp312-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.12 macOS 10.9+ x86-64

blosc2-2.6.0-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.6.0-cp311-cp311-win32.whl (1.9 MB view details)

Uploaded CPython 3.11 Windows x86

blosc2-2.6.0-cp311-cp311-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

blosc2-2.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.6.0-cp311-cp311-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.6.0-cp311-cp311-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.6.0-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.6.0-cp310-cp310-win32.whl (1.9 MB view details)

Uploaded CPython 3.10 Windows x86

blosc2-2.6.0-cp310-cp310-musllinux_1_1_x86_64.whl (4.5 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

blosc2-2.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.6.0-cp310-cp310-macosx_11_0_arm64.whl (3.5 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.6.0-cp310-cp310-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.6.0.tar.gz.

File metadata

  • Download URL: blosc2-2.6.0.tar.gz
  • Upload date:
  • Size: 5.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0.tar.gz
Algorithm Hash digest
SHA256 9e4da425632e5588595998a55ab9766e8ac1434089bf41e6dca0ac0b3706a25c
MD5 aa4247f56d305486f66884461ef9ee9a
BLAKE2b-256 26828163b22dfc7d556b4a93d4752ae43ac4ba190d0b96d1431a1e2bca5051c9

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.0-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 89b7583c4b9dfb896895adc605bebf1ff04a80455f7dabc48b826c2e1fb268b1
MD5 e6a980322cbb30b59791a50c472842ec
BLAKE2b-256 09d2167f4ba358be11def46227c2a1b39c9025a5d543d9f7d557f337d7508164

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-win32.whl.

File metadata

  • Download URL: blosc2-2.6.0-cp312-cp312-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.12, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-win32.whl
Algorithm Hash digest
SHA256 3e9a8f52fe9010c8ff5f315096dd6533fb3d1c14ece9b8590b3439aef8f729e1
MD5 8c37b5a030a0925931774b934715dcda
BLAKE2b-256 6e54eae6eca4f7a0d72311481ea9f33a6ae054db9cebd5f2b5033bb347f90667

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 f7ac178828f861f5a44257ec7a9299ae20039fdd38ada4e742f216a7680f59c8
MD5 2518968cdc8485dc1c915581e56b1210
BLAKE2b-256 16f42a9110b6da50b459f6332e4d69c4a749159e5bd3fde6aa7fca5d4119cc35

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 097976a68c26a826fd2410eece0809f7949f5d1e3625176de38dc04a0bf6dc0d
MD5 2796d1bb4ddf62fcf1f07c8623861a23
BLAKE2b-256 dfe5307547ddd2b7d129433941005a7e8960cf02b3979b9f817f369bc8d0bf0b

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 bc8152256940f89d98704efab46a1c002ccc7c6178f78b92977302f7eed06e39
MD5 a027a088102c1ff4e4d6262c77834c10
BLAKE2b-256 2976190ec473dd1a8d8d559d4770919bcadda65431cce392ce7a01472d130cae

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0304f3145f3c4e5992c2c683dd66e688172ce6544b338584a4369a38eff4093c
MD5 cfee94ab3e1649dc8153eeef23feb9bd
BLAKE2b-256 1925d2950b24817f9604e601988a89dfa873114ea54695e85129a225f0daefe2

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp312-cp312-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 6747bfb5ccafa47ef99243fda220be508b7b8324008b696046cf035d23020fe9
MD5 b8e7324be2d6d8e47edbd6840a3b17b8
BLAKE2b-256 0c41b2ade0322ed5baec8b968fe580847e1acf82bb43ff77c83ab46810abf8da

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.0-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 d9672cfbe751c7a8705b3660721d16c4c0ea857b3201a2872087c9a2bb9f87b3
MD5 a884cdb69ce0ba6a5ce5a2ab00c71607
BLAKE2b-256 6e7a76f099465b07437eeae3c64a82c3273301bfd9af8d33724c52ffeb16c9f7

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-win32.whl.

File metadata

  • Download URL: blosc2-2.6.0-cp311-cp311-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.11, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-win32.whl
Algorithm Hash digest
SHA256 c9639629bd5a0accf4b5ac8da11985aebee5fb20def78ce96037fd29a220c82c
MD5 8dfbe9a3330cde163dea773379392dc2
BLAKE2b-256 01027f9e3c6ea80b1a275c764dacd7229c5f8cfb3efecd78410e641f902e2f6a

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 3e9a2fbd36781573779b7c1d7f493bc994f4fee085d4e88a2efc3d372f03f4ce
MD5 cfabb426e10e1fb4c9b5cdfbddaa140b
BLAKE2b-256 c8e0d7487e71f478bf0ea68ab2ac8710e36eabfe1b223b62289d7b2d8e08db5d

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 960a6de48d9bf194b6458a1a59cc5f2d8e1d8e75415148a2a81d89e83593fe59
MD5 676355fe275bccb36817ae137dc87b95
BLAKE2b-256 f19a7fc548a6b88e1919608e58885aa5485e972791c58a8392bad5b3bdf23f66

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1d44812ec44fd35338da5bb72ce93f79484395f90805f36591dd1523f645fa0f
MD5 00ec1f4119e6bb58cdc7fdc7166c904e
BLAKE2b-256 57e8ae88f05e60fb3eee9946a592a28c201e6f4ac790bb0506c21b48b2a6db48

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 df7a961876ce7650d26dc66ae52567608ea9bd910746dcc44f3254d8b1bb5cc6
MD5 ca0bcba63b8b42abb8b6651c60e16fa4
BLAKE2b-256 cd3f44b68af60824fe4d921ad1f1255a92f5335c46c87529a84d1f4cc105e1a1

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 633f9dcd3b4712b1dc38a77f631fe3d8e712bc1caa1a896ef861de8fd1b07aef
MD5 c421e5ee99ab017aa711002a1d853a1b
BLAKE2b-256 6343eca77b36c4f0e7bd947586776971b6ca58d862f7cdc174c48dc542f0fe4c

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.6.0-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 89a092093ffe0b6fd535a76707c5c58cc860aca582a42cf110a8ca19c1f7ada2
MD5 6b817779c583e2031787178f65d1d5ca
BLAKE2b-256 3024727a1deb09e5095a42b171449bfd44a4a2dbe04c0f37e95b3ff5ad2094dc

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-win32.whl.

File metadata

  • Download URL: blosc2-2.6.0-cp310-cp310-win32.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-win32.whl
Algorithm Hash digest
SHA256 d7bd3ff65e8105081bf980bf830b562ea6260c1b1e12810007e513f35f80f74e
MD5 06d6954b154b4f957ebb22daa29898cf
BLAKE2b-256 f977a5f25065771603c6e6acc3fd0491cc50b1077784df66b17d7f931af77924

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 b081d5fae153f9d1ef2864155d2a49d287cc404c2a574d66c3ce06b238f48320
MD5 4155080d7185ed1c3334453d612ef3c3
BLAKE2b-256 7c9f7e072236bbea74e3e0d42980dd2b0eca5af8e02e321ac3945fa5a3b2cd81

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4be2bc0491c8aa84aa6afe9bd031712e61f384ba202ee8e8bfd79239c754e915
MD5 528f8ff7863242a47af83af31aedb8db
BLAKE2b-256 aa91ef7e0763d09bbd50822fe303aa0fc827eb5b5befd48199509ac9efc4843e

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9e09a756c5692802d063bf84e12891d4dd61bbb6493141596b3ad2a031fbf427
MD5 e15ac3edf38cb7b3beb8434368e8018a
BLAKE2b-256 689c056e8fecbd570bd17d32e4ded8705a9c68037b9453e8d8b7b12cfd2f83cc

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4142591a6bd6653961fa1f1f0f822150dd9464520f34403ae24b18201eaff324
MD5 2bc880c11a5f0bf033378744b6d33866
BLAKE2b-256 31cf0fb7d1e2aa791edbb4214fbda8f249c9f9fff0e3cee16dcdd0a9e4426e72

See more details on using hashes here.

File details

Details for the file blosc2-2.6.0-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.6.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 cc5c2da7564b6c3c5ed829cba876b586b0072ead5cdebd629ab33d3e5f4aa78a
MD5 846e984713ca5558f85a84f5d038e125
BLAKE2b-256 f046774acdf5e11c50fc5f88e90c0e4171de3d80d29b39fd279333a0574014a7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page