Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.4.tar.gz (4.7 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.4-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.4-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.4-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.4-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.4-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.4-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.4-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.4-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.4-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.4-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.4-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.4-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.4.tar.gz.

File metadata

  • Download URL: blosc2-2.2.4.tar.gz
  • Upload date:
  • Size: 4.7 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.4.tar.gz
Algorithm Hash digest
SHA256 efb85623bb2016677a6f59f53fe39ff27b3835ca1fd1542e1a6b00898a6f4123
MD5 4667ef62bd0d7b664bca6a55b7f254dc
BLAKE2b-256 21da1f9825f5c17ce8212fccbb13d8f422e2557a028f436af336a3a9b250ddb3

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.4-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.4-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 a3b0f7ff5701c81bd4650dca3ec375e57f336218ea23808b3b0a408d087513cd
MD5 9cf771120661aec55d23b1075039278e
BLAKE2b-256 46b5aed475da88dc6d5a93130d264eefff4701a152a106388721eb9169dd8545

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0674702ff600768eedbee3c1c38fdf10b2df88429fd8dda7733c9c879745514f
MD5 7939e0c47aa29f151f20020da904805d
BLAKE2b-256 f49b8c9ed843014bac6efaa2de26d4fe431ef932cb43c685293669f411fb5a4b

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 313fd8be4c6b57b4a4d312828e1df5c16e98b01e690580e7d0bc56755e49d597
MD5 ebc3dbedd42e06bdaa02ba4f381b3f27
BLAKE2b-256 2a426348c272389e14739aa996b0148b97f71e39da2be14d8b5513ec7a82ecf2

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 bf39bd1557dc8e7922c59ffd006503bf3b3159a08b367077b5f076ba4f430e70
MD5 bdbdf6dea17f2a652920db61934943e0
BLAKE2b-256 a884f56b1ba8d6ccd5557f2f8603a8391b2379a0d912fca9d46a065aecc1dcff

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 0f919918dbf59cf51fde242d95e2d794a2a45dc61dd8b45f0f50f112909d855a
MD5 d4f614dfbc965e270baba3034425b00a
BLAKE2b-256 1068eddaa549b970e73129faa1049c0ec828192c6789d6291d8ab4dec19ed2d4

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.4-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.4-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 53e7d77b6f44c820df06357ada2a6eb6411d84de9d7a306ac44ae5590c7b3586
MD5 1970867c9828ede720635a1cd4d47ad8
BLAKE2b-256 8f600139c0e7498624b659890a52c861b8751e0a08e5011631ace04aacac5a8c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6f61436249aa4d688a9c49a6f53f1a6254e41c2a90b672f9fb13d189813f6bde
MD5 bc47d603e4f6fd8a95059d1814b2925b
BLAKE2b-256 31a0a030bcd89f47ebe44fad4968220334a634493f6ff2ad3ec02b1499032175

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 cccee8442223e8583a5b06a67295e761166b6fcd817cfe236d3788426806e8a3
MD5 f1d9ba8815e1b57e4d44fef57963a195
BLAKE2b-256 921ee0f1787cadf75a1203b56e3cb2964cb4c8421367dece18062d6fe15207c3

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 014d8dddc70488ecb8c27d3022991fef3b4cb5682dbd64d9042813406c9dc562
MD5 3e9b472c8c82fc9eb28b0284a37ca262
BLAKE2b-256 d39b48c0ffd2073ee1ad25e78e5baac207f2a60c3f93087f290593467d36347f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 2dd5d471a248886f2174c5073960cc62ef979197cf5c0c6d7cecced843fd3d1a
MD5 a50501c73d2aedead7aaa287f7e9dbbd
BLAKE2b-256 ceed85d48ff2e87cff31930e3e9137d30de006e70e82ca20cfb6f77133f0f341

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.4-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.4-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 3ff1c0767d233be98182138f8ef7a7f1cdaf15fd1e047bafe20ffaf86632cd13
MD5 1ac04bf93515a4d97245081a784572b3
BLAKE2b-256 c849a45e8fa8d88a25ee3efbdf21edc7498a2de3c67577e87a49bcdc3a55fe14

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4ad1d2a7a68676e7aad713ca753ddff52f9994599661b560ac4ef1c4f257c8f6
MD5 e1050f39acdebd012caa0df00ddaba93
BLAKE2b-256 bc998a239dfa731c5512674d997842e15176fb7c16fc3685700f5a81828aab82

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 218bd0f1b5916e0b3bc92f49bb6971d9b6e5af20cf25c1ffc8fcc2994dbbf465
MD5 f14a59ef7fa057f5464de8bc55488e8c
BLAKE2b-256 59107ab9d3af68b107b1265eb180dd3f24ce0e47dde4b4ee804f49d8a047e754

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2314112a4407c2cdc933eba724282329fe43d0cc1766ee302db312fe00d78784
MD5 2699de5c66e87ea26c0a7598aad5000b
BLAKE2b-256 b16e263c01f7963b20f5d1d72fe60367e504df662523e050bab502a6115e40fa

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 2dc020f57699a98794a66dffbe5bf4b25e90d8d53efe97f777ffda1f8790a4f8
MD5 0fad29c9db3a722f5894f61b6fe40830
BLAKE2b-256 d60280d5d232c9187fe4ced7ce32d1f5124aa5dbfa1c60a85f02fca62bca947f

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.4-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.4

File hashes

Hashes for blosc2-2.2.4-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 4a9e69814dca35117a1397a619037119cfd6fc308287c765ba93d2e878b48ab6
MD5 46cb521d4b5653f8c68263c5b61141ed
BLAKE2b-256 9c217223c19c87ec9d1c8c22f185a69930f0ed7f962b84d62aecd007736dcdc9

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 50605061851b1c652ad7dd427639fd8f06f824bcd7d14651acb2cb9b35688fef
MD5 ae947c47c3dc0a07a74827a5fdd75fed
BLAKE2b-256 756490e426682e18dfac7e84b51d13098ab98c49a10637681181ff05cd237d88

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3929960c256969d7d298d2c004df1dabc68a0da8b12742af23fff41bcf5bec8f
MD5 406827f7bd697d1dda4242eea718d3ad
BLAKE2b-256 c34ef7e574f1c35f65fa4b1dc87614f78de5521a8bb755bd0410ebb604e05b67

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 7fd26a6122edf67fff8fe00365fa85effc6f846278d39d2240c9111baccd9406
MD5 4c70fafcb030f4b72733a4574ac0308c
BLAKE2b-256 1f99997ad197d0b6e55f10152939f8a0ba8164c6ed1a3bc8c3c8df21eddd6964

See more details on using hashes here.

File details

Details for the file blosc2-2.2.4-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.4-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 652219132a69aa4c76fb280824d497672e31c0652ee740095e8ebbc4527a0a40
MD5 65547eeab91a7cc9a8038cd851e44764
BLAKE2b-256 e392c02cabbe19ad790a7f3cd369bc9fe4b2387f1f99b0c81e36d156625ee986

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page