Skip to main content

Python wrapper for the C-Blosc2 library

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:

The Blosc development team

Contact:

blosc@blosc.org

Github:

https://github.com/Blosc/python-blosc2

Actions:

actions

PyPi:

version

NumFOCUS:

numfocus

Code of Conduct:

Contributor Covenant

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Compression ratio for different codecs

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Slicing a dataset in pineapple-style

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.


Enjoy data!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.2.3.tar.gz (4.2 MB view details)

Uploaded Source

Built Distributions

blosc2-2.2.3-cp311-cp311-win_amd64.whl (2.2 MB view details)

Uploaded CPython 3.11 Windows x86-64

blosc2-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.2.3-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

blosc2-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

blosc2-2.2.3-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

blosc2-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.2.3-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

blosc2-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

blosc2-2.2.3-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

blosc2-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.2.3-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

blosc2-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

blosc2-2.2.3-cp38-cp38-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

blosc2-2.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

blosc2-2.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

blosc2-2.2.3-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

blosc2-2.2.3-cp38-cp38-macosx_10_9_x86_64.whl (4.3 MB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.2.3.tar.gz.

File metadata

  • Download URL: blosc2-2.2.3.tar.gz
  • Upload date:
  • Size: 4.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.3.tar.gz
Algorithm Hash digest
SHA256 5ccd4ee21ef1e25ac54119c4a525b6bc2ba2a9a747448de1b27cf8240ce94fff
MD5 daa2c9bba017e1b26bfca5fad6bd9669
BLAKE2b-256 2137c12af4f7677a49da13444a158e2ef44989a4b6d5cb6b5b35fc17e6d07e8c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.3-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.3-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 eeb840983f1ed7bcb23526379e618ccc5a7f7c9a62ae58ef532ecd52d95e42ab
MD5 24f2b214001511d71710dc5ae17108e7
BLAKE2b-256 cfc286f307f490a153ad01edf96deba4f21264f36b3883e2ec853112869a7ab2

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2a74531d2907664120561325267124ff89a374377223170b3cf29439c93c06de
MD5 f75a9f9765d4f9ea4a9ebb1416a9056b
BLAKE2b-256 b31cec35f535207f4a0d14b181b90d63506b9126ac2a3af99b986efb69f531ba

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 77ed06591577dcf6746064b94155886be6d871fb7009e6a9ca50832a84e1f9b0
MD5 f5b55c45b8a4a23df5bf9b0484aed438
BLAKE2b-256 cc25d110e1887473ed1c5403df06bc0c68d4a44eb12dd6053994d0e32c8f5e34

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 d7bd0a91ac72004e655c67e1a42f1a214fe447d77bdf9ab381c81b35294b5752
MD5 719861fec401db2ec470ea140b595e28
BLAKE2b-256 e6e3edebb9dd03bf26fc900241b0b79185924dda858504cca2a2bb9850a410c2

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 3a1a798267e6e5c63b2f2c853789b1faf875fe07d8a0fbc50c1cf1e0150dace4
MD5 3c11920897d1c53c44f5e501d0581900
BLAKE2b-256 5bcde5ea9f4167e06e0e4116b9aa76a4717e2e68d453b18d18576ef4f636e587

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.3-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.3-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 e63942ab4fd831e37755df55cd665b6acc9b04ba5388eb4922fe34488a1ff883
MD5 e31c3a2211d39641fe3e3b5ba6313c9c
BLAKE2b-256 0ad0d95357893f4cef8409ca040deafa069bc6af87a9cceffecfe8ff1808be3b

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1a418c1a72b9a71ccca9b71c44eebb491b3287b7ed38a82a24684b7f2d38a2c7
MD5 5a9db46fbd30580c763173827f6ed5da
BLAKE2b-256 0cadb8ba993078fdc83119081b4290bda160a5b074048169f19785d4091a961a

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c33066f1f70431e252c0e0f8c7743112c73d2442e78b63abcf68de779c60f9f9
MD5 3ec7f9b5b2ee8983e6dfbfc66aadcda6
BLAKE2b-256 fcf5d2ee6bcfda5627ca82e2cc6db33127f73fe8fa8817ae5251ff3a9956cc64

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4828c6886cff6dc51e3cc3e8e21f24ccbc58ff03b481c534a51461e380207105
MD5 efe4626ce58a9af2a1c29b6e46c3e1f5
BLAKE2b-256 9654f570d33ca4ec145ef13d84f16305767471b2a5995d0608fc8087cec4146d

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 0b9bc5dfa19081b2f4ec856181af99fc9c438509e4102370ae0ec9756630fd5f
MD5 f036eea5c2f467afa92bb6ac7029a604
BLAKE2b-256 f0c891b6a42200441e0c632d43b49a5be6e9e3b3715cb84bbf18c7a90d18bd21

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.3-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.3-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 cf97e16ba60ba0fcee2aff7314888ee7c03d451cafe3f799d322b0e6261dc8ba
MD5 848459efe385b98f90b71dfa2a435df6
BLAKE2b-256 c4642377162ea17dfe04a277215878e507e6ee158fc74aecb17eb20f0534135c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e24bddd44402ba91035db594aba0e03d53e65db418f959549ca6dbb5b48200f1
MD5 b3db8273865de82605e4ca6bd7366610
BLAKE2b-256 c5267572cb5b5fabb137c9b769a73a338892b1df8d0c2060542c56b4090405b9

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1a394c6ff4d6e7b10c60a50e1ce94e2fcf8be9b8caa6124db5983bd107e94ae4
MD5 94b8c23dc0ab1647ce948f426e03dd48
BLAKE2b-256 cb677f7231497f81db93d610c9c1da3060fdfe9dcedbccea601c5a1b27a0074c

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f4902026d44826c0b725963098e1a3c29abfb679c776a0f2f875488316715f8a
MD5 e6ec321819ef1d451f3fc9ebb9db0476
BLAKE2b-256 39e666e40e0b7cb2bb66b3f09e72a10af3f2a9ce2b045063e4c0977e4f9d81b2

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 6078dc80cac9aa59e3d8aad2b4ef88b8e798c4f5531fa11d613aaa7945920821
MD5 98d4963d7948400ac88ba377c3eb628b
BLAKE2b-256 894bf7c8ca526fe10b54384007a612138e52e26b49d27ef53684baf3bcc05300

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blosc2-2.2.3-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.3

File hashes

Hashes for blosc2-2.2.3-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 ddacf096140e515032cb97a0512189183a57a564acc356a8299669bbb2905619
MD5 3b8793ee9b282492e43a67d94fc58d32
BLAKE2b-256 5c9e04ff2aa8f1e38f5fd7fc6b33eeafc6be2f435863b6f1af7f5383cb4b4e02

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b470af7e42e5fe1b122a4859086a7447c49808c36c355dc2ed4121b8be511f6d
MD5 8631967482636bd26e40aae37face1a3
BLAKE2b-256 c1390e95cff858b2c97540c7d0f159da6550e1503cca0dbe0178f5076d104d12

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 30c32a36ce83a37d6e2c14c1eda4ea0382f4358bcacfc243a7fa39a54661ade8
MD5 26db2f67b3bc811960b7b93f1f140a4d
BLAKE2b-256 0500f29ebb5f376f305e063f0357240043c80696860fd88317658133f5b6cf0e

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5e5cde5b5f2a4bd92e0ff6d381e89b367e3cb61c702e01ab264536339592d6f7
MD5 b18c0c712fa7070eb3810b483330d23e
BLAKE2b-256 8f9817306c38676b4d8a6a789fced9a4765e24c8ef3e546a5a9c4a0fc8c07f74

See more details on using hashes here.

File details

Details for the file blosc2-2.2.3-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for blosc2-2.2.3-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 0020580e4899d7c7573acb92a06d895f783e9033ebc53991d458e7ac7515ed04
MD5 be5d880113da41f57a8bddab833d241b
BLAKE2b-256 77d2ded29ede9927e3371a5dfbc002a2a9fe6c604b47b654b11debd2e0acf490

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page