blosc2

Python wrapper for the C-Blosc2 library

These details have not been verified by PyPI

Project links

Reason this release was yanked:

Not working because AVX512

Project description

A Python wrapper for the extremely fast Blosc2 compression library

Author:: The Blosc development team
Contact:: blosc@blosc.org
Github:: https://github.com/Blosc/python-blosc2
Actions:
PyPi:
NumFOCUS:
Code of Conduct:

What it is

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.

Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.

In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.

Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).

SChunk: a 64-bit compressed store

SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.

Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:

Compression speed for different codecs

Decompression speed for different codecs

while reaching excellent compression ratios:

Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):

Compression speed for different codecs on Apple M1

Decompression speed for different codecs on Apple M1

Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements

NDArray: an N-Dimensional store

One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:

https://github.com/Blosc/python-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true

To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:

https://github.com/Blosc/python-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:

Installing

Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:

pip install blosc2

Documentation

The documentation is here:

https://blosc.org/python-blosc2/python-blosc2.html

Also, some examples are available on:

https://github.com/Blosc/python-blosc2/tree/main/examples

Building from sources

python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:

git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace

That’s all. You can proceed with testing section now.

Testing

After compiling, you can quickly check that the package is sane by running the tests:

python -m pip install -r requirements-tests.txt
python -m pytest  (add -v for verbose mode)

Benchmarking

If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:

PYTHONPATH=. python bench/pack_compress.py

License

The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.

Mailing list

Discussion about this module is welcome in the Blosc list:

blosc@googlegroups.com

https://groups.google.es/group/blosc

Twitter

Please follow @Blosc2 to get informed about the latest developments.

Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

@ONLINE{blosc,
  author = {{Blosc Development Team}},
  title = "{A fast, compressed and persistent data store library}",
  year = {2009-2023},
  note = {https://blosc.org}
}

Enjoy!

Project details

These details have not been verified by PyPI

Project links

Release history Release notifications | RSS feed

3.0.0b4 pre-release

Oct 2, 2024

3.0.0b3 pre-release

Aug 29, 2024

3.0.0b1 pre-release

Jun 21, 2024

2.7.1

Jul 30, 2024

2.7.0

Jun 20, 2024

2.6.2

Apr 6, 2024

2.6.1

Apr 4, 2024

2.6.0

Apr 1, 2024

2.5.1

Jan 25, 2024

2.5.0 yanked

Jan 24, 2024

Reason this release was yanked:

Introduced a dependency on blosc2-private.h

2.4.0

Dec 28, 2023

2.3.2

Dec 1, 2023

2.3.1

Nov 8, 2023

This version

2.3.0 yanked

Nov 8, 2023

Reason this release was yanked:

Not working because AVX512

2.2.9

Oct 5, 2023

2.2.8

Sep 27, 2023

2.2.7

Sep 13, 2023

2.2.6

Aug 1, 2023

2.2.5

Jul 4, 2023

2.2.4

Jun 21, 2023

2.2.3

May 18, 2023

2.2.2

May 11, 2023

2.2.1

May 10, 2023

2.2.0

Apr 4, 2023

2.1.1

Feb 24, 2023

2.1.0

Feb 24, 2023

2.0.0

Dec 20, 2022

0.6.6

Dec 15, 2022

0.6.5

Dec 10, 2022

0.6.4

Dec 9, 2022

0.6.3

Dec 9, 2022

0.6.2

Dec 8, 2022

0.6.1

Nov 30, 2022

0.5.2

Oct 23, 2022

0.5.1

Oct 12, 2022

0.4.1

Oct 3, 2022

0.4.0

Oct 1, 2022

0.3.2

Aug 24, 2022

0.3.1

Aug 18, 2022

0.3.0

Jul 6, 2022

0.2.0

Oct 7, 2021

0.1.10

Jul 10, 2021

0.1.9

Jun 29, 2021

0.1.8

Jun 22, 2021

0.1.7

May 26, 2021

0.1.6

May 26, 2021

0.1.5

May 14, 2021

0.1.4

May 14, 2021

0.1.3

May 14, 2021

0.1.2

May 14, 2021

0.1.1

May 10, 2021

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

blosc2-2.3.0.tar.gz (4.7 MB view details)

Uploaded Nov 8, 2023 Source

Built Distributions

blosc2-2.3.0-cp312-cp312-win_amd64.whl (2.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 Windows x86-64

blosc2-2.3.0-cp312-cp312-win32.whl (1.9 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 Windows x86

blosc2-2.3.0-cp312-cp312-musllinux_1_1_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 musllinux: musl 1.1+ x86-64

blosc2-2.3.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 manylinux: glibc 2.17+ x86-64

blosc2-2.3.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 manylinux: glibc 2.17+ ARM64

blosc2-2.3.0-cp312-cp312-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 macOS 11.0+ ARM64

blosc2-2.3.0-cp312-cp312-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.12 macOS 10.9+ x86-64

blosc2-2.3.0-cp311-cp311-win_amd64.whl (2.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 Windows x86-64

blosc2-2.3.0-cp311-cp311-win32.whl (1.9 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 Windows x86

blosc2-2.3.0-cp311-cp311-musllinux_1_1_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 musllinux: musl 1.1+ x86-64

blosc2-2.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 manylinux: glibc 2.17+ x86-64

blosc2-2.3.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 manylinux: glibc 2.17+ ARM64

blosc2-2.3.0-cp311-cp311-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 macOS 11.0+ ARM64

blosc2-2.3.0-cp311-cp311-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.11 macOS 10.9+ x86-64

blosc2-2.3.0-cp310-cp310-win_amd64.whl (2.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 Windows x86-64

blosc2-2.3.0-cp310-cp310-win32.whl (1.9 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 Windows x86

blosc2-2.3.0-cp310-cp310-musllinux_1_1_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 musllinux: musl 1.1+ x86-64

blosc2-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 manylinux: glibc 2.17+ x86-64

blosc2-2.3.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 manylinux: glibc 2.17+ ARM64

blosc2-2.3.0-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 macOS 11.0+ ARM64

blosc2-2.3.0-cp310-cp310-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.10 macOS 10.9+ x86-64

blosc2-2.3.0-cp39-cp39-win_amd64.whl (2.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 Windows x86-64

blosc2-2.3.0-cp39-cp39-win32.whl (1.9 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 Windows x86

blosc2-2.3.0-cp39-cp39-musllinux_1_1_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 musllinux: musl 1.1+ x86-64

blosc2-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 manylinux: glibc 2.17+ x86-64

blosc2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (4.2 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 manylinux: glibc 2.17+ ARM64

blosc2-2.3.0-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 macOS 11.0+ ARM64

blosc2-2.3.0-cp39-cp39-macosx_10_9_x86_64.whl (4.4 MB view details)

Uploaded Nov 8, 2023 CPython 3.9 macOS 10.9+ x86-64

File details

Details for the file blosc2-2.3.0.tar.gz.

File metadata

Download URL: blosc2-2.3.0.tar.gz
Upload date: Nov 8, 2023
Size: 4.7 MB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for blosc2-2.3.0.tar.gz
Algorithm	Hash digest
SHA256	`2a1c664915b8226d602f03767fe0c8228232d0327546068e8952e1fbc2fd25f7`
MD5	`9fa984057033b0c787e0c00e4b22c970`
BLAKE2b-256	`58a8ff71cc6bcdcfa8003791b0f4db56641b197e9e24ccd4a66c83e65f16e57d`

Algorithm	Hash digest
SHA256	`7c30836a3bd2de31577e5f82ed77d0ce07bf7ce95da59cfb7c160bad1da79a86`
MD5	`cabcae51f4e2a4c6ed2111182a214d47`
BLAKE2b-256	`8e7b2b79d53aaece6e78a24d0e644aaa1c3729d756209de2990ace7ab6dd210f`

Algorithm	Hash digest
SHA256	`c8110d2f12337fd68c9a00ab3708ae3b394224e131c39461d8ef76d11a85f310`
MD5	`b948a1873b56bfb24627a41e28e5486c`
BLAKE2b-256	`fe038766794a2c779047e863d475df01fddf7803affa2346a3c8b1d3eb685bc1`

Algorithm	Hash digest
SHA256	`d8f161690474ed16afd7e71d5b35db051dce1c1e0bdc61f8e78f6084e0a68e99`
MD5	`91b101c5490948fbc1e67d7e58300472`
BLAKE2b-256	`b9633d1f49fcfea9d2700c61fde53de3cb8a1097ab37b63ab8ecefc808a8825a`

Algorithm	Hash digest
SHA256	`e6d9ec4bad030a08ed366fde7448456c949e3f4287686ffd8f394c11994f5dbf`
MD5	`62041d348a28c5a8aebe142f484393f8`
BLAKE2b-256	`71aadc5fbb3dc67be83b9df2496d9a7c0cb48aca7cfb1fb4e93a123bbdc8f460`

Algorithm	Hash digest
SHA256	`37caf93bd6580b0bd80ceb22ce15b81fe085e1238a0dab57040ba371cfa45602`
MD5	`da03741b49db994f645de7ccbe3e4732`
BLAKE2b-256	`246ab2588c734b2b78e2319e44a40fdd4e640385ec98b3da25d3dc7009c862b4`

Algorithm	Hash digest
SHA256	`fb5c16a888757689a967deeddbca67bfc587e23efaf2cf12c38ee530ba071452`
MD5	`339ec4bea0811f23a253fe880456d574`
BLAKE2b-256	`960b7ab3a5e8142f4e0cc632d4df6393276c68a06a849c27d75324ed748f3b34`

Algorithm	Hash digest
SHA256	`4fb9b1fb512be4a82fc17dc47d8d6fe5c8ab49abd609e29bd6a2fd7b151381e4`
MD5	`46889dc32214ea6cb1c2e37b21c7d50e`
BLAKE2b-256	`ce2c36f009dbacb2a9162a12fb104b360aa6e491fe988838ad2408b21a837e8e`

Algorithm	Hash digest
SHA256	`cfedb48c5ebdf22f40e2ed2332cc4bd0b13cf2ae9e9254070d9249065dff7040`
MD5	`ace52116f61c494ca956054880a424af`
BLAKE2b-256	`73fd1e8352b7778056ce454160fdba9eda462e30ca47ee8e1e37ce36585c3f2c`

Algorithm	Hash digest
SHA256	`cfc5f8c0dc040445c2205da63dcc8c656aac671e04dcee02b566434c5a175ac6`
MD5	`4fd7b427d79de8a4145222ae938cb416`
BLAKE2b-256	`ac9419538804e191b227f28ab71a76fa80e16b9cccc39c8c8ba38d1ae338b795`

Algorithm	Hash digest
SHA256	`2dccfa9932e4ab2037122110e64aa9f3b213044821d2b1cc58477c7585c8077c`
MD5	`3651f18c516997efdace46599bdb6fff`
BLAKE2b-256	`390864ac3227c3e912ab3d295429eb5c072a2d765c65fde0cd29ee1ab6072248`

Algorithm	Hash digest
SHA256	`bb37e557f7c670893aae304082ab021eb3a9d528efa775901cf340a9fdb7068a`
MD5	`8c1994e264bfe0c9118abb21d33f6279`
BLAKE2b-256	`6f1db03ede7b5eb3a03fdb491bff9c262df36faf6f515bc288f2e13dec429c3f`

Algorithm	Hash digest
SHA256	`a98155b3f2880c33e6ffd5aca9ec08d00f9b3e1451a0e31e031f1c1c04cf0ed4`
MD5	`8477d643b57dbc907690da182ada11a9`
BLAKE2b-256	`64ce93c8f2c0d206ab54c671916b300418965edf09eada773fbd20a26eb96212`

Algorithm	Hash digest
SHA256	`55a836e5a7055929bb0a9bf7a2794a3a87282d6e222327a32fe1f85d1d1fe322`
MD5	`f8520132ba64b1e5f8fbb1cb81f71667`
BLAKE2b-256	`0963006d68ab59252f1bc7c088e4356c54bebe7611677947ca462303605f943b`

Algorithm	Hash digest
SHA256	`2841b866615f4e92b4051b17c5d28123ce0501f7c5f34ddbdda452dccf19c405`
MD5	`cd8c1265e3af605c8511599f78267186`
BLAKE2b-256	`240c9525b45018c351005a6a5512b99631a4fb5f0fbecdd7f60818be50664bfa`

Algorithm	Hash digest
SHA256	`430a9b175c88598844271d67bac92a077b59b6c96955170b3a7b16f634b4bd56`
MD5	`3571c9b2e70d3f7d40bbbf988b5b41e9`
BLAKE2b-256	`84b15fcf0a265a02bf8c928fa09bab4875cb1bbbd9acae05bfd8b9424a21de7b`

Algorithm	Hash digest
SHA256	`de4ddce996c6d73f7bf6c172c5e6896a11708c6a0ce3e61dbaf7ba6ba4d055b7`
MD5	`19ff7eba27b23c89594a66da3eddb7ec`
BLAKE2b-256	`41ff7ce9ba3425d5e4cd265b3b838d4efbfcbd6074bacebcc1dc604a07be2f53`

Algorithm	Hash digest
SHA256	`d10c41d5f7a7c8bb1ce06bef91b4f2728bd6488c188cfeab83c5e0815a8f6ff2`
MD5	`39df07ef8fad740898cc8d2e9ea35758`
BLAKE2b-256	`db5d2c46e756ac8486f790253ede33895801bbc455c376df3bb87fb7efd47bb9`

Algorithm	Hash digest
SHA256	`e489c5356974f84b943c78f3591973069f003ea92a2cdf77cf4a20518b7e36f5`
MD5	`a16fef480e9ccb7864d9f7761b4cb9f3`
BLAKE2b-256	`1b39500beb13f51f29d9dcb23fa17adb6e3bff682868eadd25bf4a48405942b0`

Algorithm	Hash digest
SHA256	`3d6a5b12d005d6597e85d0d0b953eb441d660045fb56e4188a24ed9729330709`
MD5	`ca07ab6fd67667cbe03f2501eb6ce51f`
BLAKE2b-256	`c0604944e43e60020a54803dd9be1ef0b5ec5ff941c12cd5c652490e90f90179`

Algorithm	Hash digest
SHA256	`e379249f7f50e9544cbc1c312f1317402ced90f7c6f94b00c023fc55fea83409`
MD5	`2ff3981139bde7b7d5a7ae84be8a7f71`
BLAKE2b-256	`4042c7088a9108d4d24b6e01545232a572e144123a010e460ed3123a469ab9fc`

blosc2 2.3.0

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Project description

A Python wrapper for the extremely fast Blosc2 compression library

What it is

SChunk: a 64-bit compressed store

NDArray: an N-Dimensional store

Installing

Documentation

Building from sources

Testing

Benchmarking

License

Mailing list

Twitter

Citing Blosc

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

Algorithm	Hash digest
SHA256	`39cfda02e9624e04d695880b3f1d02cd70eca8ba5847443c57fb857e2dfebd65`
MD5	`90d27d69d9e2c6451c9fc3929e2d8960`
BLAKE2b-256	`007d957b839017c1198f9fb01acdab1d14e7f8199277a4fc2f7d92f7a7cf8d2c`

Algorithm	Hash digest
SHA256	`97d32972a97ada1308ccabe788569f7c7f3470afce781720b5e4077550a42d01`
MD5	`4a14308d03cadec7874282e339606088`
BLAKE2b-256	`415c9053fc053d8edb358cc7c753d443fb7f49b5f64c0dd78f5cf9f8cac74cb9`

Algorithm	Hash digest
SHA256	`b7c191a7ff1d365ef9606f126c639dbe4052793535b1217329d4d7e8c69ace4e`
MD5	`30d59837dfdd70d57347f628cc0b232e`
BLAKE2b-256	`4353a4885f443666d952cd5ad05fd66215beeda8d787b9767b921e03e2d1f885`

Algorithm	Hash digest
SHA256	`15d7b1c630b983a6baa682045d7d813e89b1a05e0c83d9b541a84a8a3d060b30`
MD5	`68cda38bb88bf05511a6160e1797c50b`
BLAKE2b-256	`d7ac46b6060039d150560a5d87b0a4584ac64a90610c5b307d2c29e12bcb642e`

Algorithm	Hash digest
SHA256	`4f69e2aa30dbfa9ae830c62bc0dcdc0fa0240005f03271d2daf38bc6bfc45150`
MD5	`4001235e226ccd0ae6454ba5b599e823`
BLAKE2b-256	`59f74ef0af7cfdd54a3b4aa2a813af440c1f4130bf66d1e2d40f788cbca3d873`

Algorithm	Hash digest
SHA256	`d914071df65b62bd363f1f7833784447df4b39611eb972c9d770ce26ad0c8199`
MD5	`e4934fb88c65fec7e3c13e0cffe77253`
BLAKE2b-256	`2c15a01d5bedbf86cacad7f81809b85ea9bb42c01cc2e826f639e865fae15e8c`

Algorithm	Hash digest
SHA256	`fa7fcbffb7b3d462cb91328464457f76888c1e68bcd1a256837419e2e8a1eb44`
MD5	`9b49146e3a2bbed84df786588944c52b`
BLAKE2b-256	`2f447a1bd07fd090c677c7054e3ba8a632f35c030b61f050b3578690b6ed1a1b`

Algorithm	Hash digest
SHA256	`48c4271651ec58c2e72fab13e4a59f7ac795dec833501d061eb3a340dbfb725c`
MD5	`3f04ecdda082f36b625253a45ad22f5c`
BLAKE2b-256	`0893c8b67a925582122fab1e3b9f0a6d55878377c0e999de771678f727e18e1e`