This library provides a simple Python interface for implementing erasure codes. To obtain the best possible performance, the underlying erasure code algorithms are written in C.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

This library provides a simple Python interface for implementing erasure codes and is known to work with Python 2.7 and 3.5 through 3.13. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.

PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3], Intel’s ISA-L [4] and Phazr.IO’s libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010[2]). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).

Installation

Install pre-requisites:

  • Python 2.7 or 3.5+ (including development packages), argparse, setuptools

  • liberasurecode v1.4.0 or greater [3]

  • Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4], etc

Install dependencies:

Debian/Ubuntu hosts:

$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirements.txt

RHEL/CentOS hosts:

$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
$ sudo pip install -U bindep -r test-requirements.txt
$ tools/test-setup.sh

If you want to confirm all dependency packages installed successfully, try:

$ sudo bindep -f bindep.txt

For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.

Install PyECLib:

$ sudo python setup.py install

Run test suite included:

$ ./.unittests

If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to:

/etc/ld.so.conf

and then make sure to run:

$ sudo ldconfig

Getting started

Examples of using PyECLib are provided in the “tools” directory:

Command-line encoder:

tools/pyeclib_encode.py

Command-line decoder:

tools/pyeclib_decode.py

Utility to determine what is needed to reconstruct missing fragments:

tools/pyeclib_fragments_needed.py

A configuration utility to help compare available EC schemes in terms of performance and redundancy:

tools/pyeclib_conf_tool.py

PyEClib initialization:

ec_driver = ECDriver(k=<num_encoded_data_fragments>,
                     m=<num_encoded_parity_fragments>,
                     ec_type=<ec_scheme>))

Supported ec_type values:

  • liberasurecode_rs_vand => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]

  • jerasure_rs_vand => Vandermonde Reed-Solomon encoding, based on Jerasure [1]

  • jerasure_rs_cauchy => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]

  • flat_xor_hd_3, flat_xor_hd_4 => Flat-XOR based HD combination codes, liberasurecode [3]

  • isa_l_rs_vand => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]

  • isa_l_rs_cauchy => Cauchy Reed-Solomon encoding (ISA-L variant) [4]

  • shss => NTT Lab Japan’s Erasure Coding Library [5]

  • libphazr => Phazr.IO’s erasure code library with built-in privacy [6]

Code Maintenance

This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.

References

[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org

[2] Greenan, Kevin M et al, “Flat XOR-based erasure codes in storage systems”, http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf

[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode

[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version

[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, “NTT SHSS Erasure Coding backend”

[6] Jim Cheung <support@phazr.io>, “Phazr.IO libphazr erasure code backend with built-in privacy”

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.6.4

This version

1.6.3

1.6.2

1.6.2rc1 pre-release

1.6.1

1.6.0

1.5.0

1.4.0

1.3.1

1.2.0

1.1.1

1.1.0

1.0.9

1.0.7

1.0.6

1.0.4

1.0.3

1.0.2

1.0.1

1.0c2 pre-release

1.0c1 pre-release

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.2

0.9.1

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.17

0.1.16

0.1.15

0.1.14

0.1.12

0.1.11

0.1.10

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyeclib-1.6.3.tar.gz (50.9 kB view details)

Uploaded Source

Built Distributions

pyeclib-1.6.3-cp35-abi3-musllinux_1_1_x86_64.whl (511.0 kB view details)

Uploaded CPython 3.5+ musllinux: musl 1.1+ x86-64

pyeclib-1.6.3-cp35-abi3-musllinux_1_1_aarch64.whl (471.5 kB view details)

Uploaded CPython 3.5+ musllinux: musl 1.1+ ARM64

pyeclib-1.6.3-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (492.3 kB view details)

Uploaded CPython 3.5+ manylinux: glibc 2.17+ ARM64

pyeclib-1.6.3-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (532.5 kB view details)

Uploaded CPython 3.5+ manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

pyeclib-1.6.3-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (531.6 kB view details)

Uploaded CPython 2.7mu manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

pyeclib-1.6.3-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (531.6 kB view details)

Uploaded CPython 2.7m manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

File details

Details for the file pyeclib-1.6.3.tar.gz.

File metadata

  • Download URL: pyeclib-1.6.3.tar.gz
  • Upload date:
  • Size: 50.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for pyeclib-1.6.3.tar.gz
Algorithm Hash digest
SHA256 396004d596f6006b3f13ed8a403049f7a9b338f47142da10aa1bcefe978c7989
MD5 8ccc7052f29811ac7dfd58ec6433e13f
BLAKE2b-256 2eb9a3b1aeb2dd4d8d2b6a8ab2a2c1e32edd43745ac0382d0a2ba330e9c3e450

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.3-cp35-abi3-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.3-cp35-abi3-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 d88ba8b2a51bf242da67eb4dfc68c9efc9e422c5980c7ec29632fca64779da37
MD5 a9b3a30d3472f6c141602a933d321aba
BLAKE2b-256 578035ff6d25107611b0498c54902c3e3d1fc56937f047d41b01e73d621333b3

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.3-cp35-abi3-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.3-cp35-abi3-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 a8aa8afc4f55d053da8f567591ec60b1264f948264b01cef813bc75ef91913b0
MD5 45cb3cf52ea7e00953fec29a6c1bf5e3
BLAKE2b-256 c337ae87537271004673f813f46938a70b63fc9be627c7b13e0d4e75a774df36

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.3-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.3-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 641969dc8ab66f73474d513afd9f21897762fb6e8e9270afc2cc086a135d47da
MD5 acebe07f4ec088cbb3ce32076800319b
BLAKE2b-256 1a60936de4b00173ae0d6d76674ac70475229d92b341e40b6844ca6a3be09ab2

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.3-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.3-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 fd569571392f48d863648f6c3cf305858145ca0b8d7bb10fe0bc0f5f3710f026
MD5 b2481ea926a0517041b1ea5365d6adcc
BLAKE2b-256 2548ca3a1a2ea5793fc4e49313d09029bb016b2d5ed2cbad1ae5e216429f6332

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.3-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.3-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 4b03c3f59b2d3c5d2dd27de622ef7644399c0b54be06bfd76f706175e6fc2145
MD5 510f44f9d86e545813378cbfd38e7469
BLAKE2b-256 8e95e002c6d82644f81bca1c6ad6d3e6b72e872b0ee8135f1d98c3e8645df932

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.3-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.3-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 ab06cc700ba2805c04d034316298006bf920da12ba34ae3777e063648887774e
MD5 c250c90a30d96fffe807dc5866324d8e
BLAKE2b-256 76f8c4e1de64b6210ce1c78706c1a641b023ee9d9e42a5c4ad149b56b8869c90

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page