Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.5.tar.gz (4.7 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.5-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.5-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.5-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.5-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.5-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.5-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.5-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.5-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.5-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.5-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.5-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.5-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.5.tar.gz.

File metadata

  • Download URL: blosc2-2.2.5.tar.gz
  • Upload date:
  • Size: 4.7 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.5.tar.gz
Algorithm Hash digest
SHA256 b8c0940267dd8df7a2b95ee2a3701ded2d240c85b27514909d9997425a091f54
MD5 230961cfe4e50472cab2290edaa953e6
BLAKE2b-256 a9eb899d65118ef3ae6d996a488c59ea1020e366d682f7ad2dc2d2f7f41c4848

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.5-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.5-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 78eca62700347452d5d4f7c0f9a789811d17a35709c0b181d93eb4f44bb93600
MD5 1e167b3236b16ad765c8974239273dbd
BLAKE2b-256 dc516746898e109cc730bce62f5c600b9bc1d250cdd7f2a5f2a7b4211f5a2f6f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 cea84fa5ba603896b522afd782e42cf8adbb93dfcb4ff5349e1ba2f114794676
MD5 0fcd25e83c2467d995f8a7a89998a27f
BLAKE2b-256 8b2967231597205620f932cfe3d1b09f279de5229f16c0343f4ecd2add4edb8c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 4a786d7984e51a132fa88ac1a2a429585a37bf4d2cf5e895cfc9013e5eb3ef39
MD5 62ab44cf535f8f0a638db5d8f7efdb93
BLAKE2b-256 91c9c68454664d278668c274503f327e2cc93c16451ecb3022cc9eb0cef1f3e9

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2d8a481a018241395f30c96745047448c51d36b9437f00f7c1913dfecd586443
MD5 f66829be12a3ed6f22a28a8f3d8e1bb3
BLAKE2b-256 34cfec87aaa48687cbfc1f09bc3aed051ff846e81170cabd19c8456f6cb4ea3a

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 4715fab746a7cd3815d3117805e43c58e84f732db376d698245fd1181ff68eae
MD5 b1b05dbe42bcb69d2516abb7673f53b0
BLAKE2b-256 4ba9d1b8651a0c912f273351c2528270502a343440c6e3d300f92f42379783cd

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.5-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.5-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 d91a4a6fdad9d93b958440231255874fccbeb346faeaf6d0468f308e1025024f
MD5 479c6cf0a44783aae5fa0a2baca239ea
BLAKE2b-256 34f63015302e4156217c6d4197c21617ca2c830dfe28a229280590a75a468622

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 affcbdcd88d1889bdf633831f17c7a47700fc6030416b562bbb00a6e0f075d39
MD5 25031233ae675bb178a2799f6733042d
BLAKE2b-256 4104c646249d6040ab03cc2b9d15f0406f401b9b06b23ca51e846acc08cb5854

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 99277cf357f2a6ae0253d44de40bc5aa38f5ce643e758a54b41ead12df94a633
MD5 2c7aa5ad1a2749510efc78feaab813f9
BLAKE2b-256 b8d0c1d7a78865964d4885c3fd4be66bbefe34af2dd0ebcef45801c5685c2a4d

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 eeed99519e1bf62d8de565b27f1a4ec12ced6fe7d0b2c632d8e8f4991577529f
MD5 a60cb1000c891b7f7961b65d3acfddd6
BLAKE2b-256 81fd5717d838a635654bd12f7514e1a8a3dffa55ba27a51fe223f69738c4699c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 cdfc93016944d1902d725c2e2615850c0ea3d4fca86497c77a64974e3280b5a2
MD5 5a29f20733ed8117c6135b22a4ae2084
BLAKE2b-256 92d242a6411b12ac1988554f671b351b42afa75b6be3adf256335adbf7f88471

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.5-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.5-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 7c09665f43545716bab222fcf3b8e763d116dc7002241ba0ddaeb83bb4d14642
MD5 34faf261e183fb2e0f7906624a1db448
BLAKE2b-256 6666ce24d402d9a2e8529eba7664f08387abb8fbc80fc421ef47c15b28b5a534

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a3e64f316309c451d96d3e4838a94b74bd956c87aa8407af4b7d63495f34722e
MD5 7afc73a937fb52b0a3153d7bd58656ca
BLAKE2b-256 2b80124ba450cdefe573fb033da3b2c9d4b48ca39ce2334a0e5642af935d1dc5

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ee3a18b844bee29f67f0390d284d4aaa09128f35aa373a73c34542f685a35a52
MD5 0c9de87132be0b6caa64b9835dc3e753
BLAKE2b-256 d0f0fafc4fdda2a21578cf3930888bd1a6ebc602a739fce98d66d129b336e2fb

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4a99374880bc95fad272cd5362727a4c7ebff7fdce154aa403c82afa88653a25
MD5 7030f03adf418c4fbeb3eae8a1063297
BLAKE2b-256 b5b56c5cb4df313ed5de8d7ab9bb9bd332f873fd9ec87fdda4156785a1839422

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 4dbca224653ca0be5a9b245d1cf39d157809bc22c0ba9e1bcc93a4199fc3ca27
MD5 67f0d7db407e62ee5f882163d1f852ae
BLAKE2b-256 91ebae093d90cde453702b7c1feb54a51639f8a908055b79fc01e478d0f01a67

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.5-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 5eb8e17c776a4bc51bc06f5f01d35bfc2489115d0a8b78223a265370694f6295
MD5 f3e92007a1c5800e522fade462801ce5
BLAKE2b-256 bcc6c0499df033b044340e91df7c0678a79490832b9a357f9f57e621fb6a2d23

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 21a03fdb092089c64dbb5d080b9f3b1ab2b769a7a46dc6e4bdf580dc95d3c268
MD5 8a87f3b11c828c930cfcaa34a424feee
BLAKE2b-256 5d003732e0a38552d75819d5aab3551ea1aa5c2c3078447c485358538e952aef

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 270a465794b3d44abcac12bd4d3883a43846c782f9d2672e533df88ffa0f655a
MD5 70c21b6abc3bfc4e74106268a4d8c1c6
BLAKE2b-256 e8194627fcdb3a9fd11da7b9dddd489223eb197820a854f016d0dddcad1d45e3

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 80cb1d62196cc0dfac873f763e7102a9cd0f18f5ea190e072a429d19eb3ea679
MD5 8fa868e58ec9f1c8dca422ac8d8e06e1
BLAKE2b-256 444256e3811e47f3fc16e5a24e3aa99011d8a80059b3eaf38912062e48daebc5

See more details on using hashes here.

File details

Details for the file blosc2-2.2.5-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.5-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 9f4882e97f810d8f30918140275cc1b38f204c54778ba5a41e94f1145d998482
MD5 90600d595bea0567a6b70a6324372d7d
BLAKE2b-256 6aa1ad148fde02d1b60aec0f09d81aea8e7fe5ff34513a94a31dc24b62337083

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page