This library provides a simple Python interface for implementing erasure codes. To obtain the best possible performance, the underlying erasure code algorithms are written in C.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

This library provides a simple Python interface for implementing erasure codes and is known to work with Python 2.7 and 3.5 through 3.12. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.

PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3], Intel’s ISA-L [4] and Phazr.IO’s libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010[2]). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).

Installation

Install pre-requisites:

  • Python 2.7 or 3.5+ (including development packages), argparse, setuptools

  • liberasurecode v1.4.0 or greater [3]

  • Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4], etc

Install dependencies:

Debian/Ubuntu hosts:

$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirements.txt

RHEL/CentOS hosts:

$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
$ sudo pip install -U bindep -r test-requirements.txt
$ tools/test-setup.sh

If you want to confirm all dependency packages installed successfully, try:

$ sudo bindep -f bindep.txt

For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.

Install PyECLib:

$ sudo python setup.py install

Run test suite included:

$ ./.unittests

If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to:

/etc/ld.so.conf

and then make sure to run:

$ sudo ldconfig

Getting started

Examples of using PyECLib are provided in the “tools” directory:

Command-line encoder:

tools/pyeclib_encode.py

Command-line decoder:

tools/pyeclib_decode.py

Utility to determine what is needed to reconstruct missing fragments:

tools/pyeclib_fragments_needed.py

A configuration utility to help compare available EC schemes in terms of performance and redundancy:

tools/pyeclib_conf_tool.py

PyEClib initialization:

ec_driver = ECDriver(k=<num_encoded_data_fragments>,
                     m=<num_encoded_parity_fragments>,
                     ec_type=<ec_scheme>))

Supported ec_type values:

  • liberasurecode_rs_vand => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]

  • jerasure_rs_vand => Vandermonde Reed-Solomon encoding, based on Jerasure [1]

  • jerasure_rs_cauchy => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]

  • flat_xor_hd_3, flat_xor_hd_4 => Flat-XOR based HD combination codes, liberasurecode [3]

  • isa_l_rs_vand => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]

  • isa_l_rs_cauchy => Cauchy Reed-Solomon encoding (ISA-L variant) [4]

  • shss => NTT Lab Japan’s Erasure Coding Library [5]

  • libphazr => Phazr.IO’s erasure code library with built-in privacy [6]

Code Maintenance

This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.

References

[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org

[2] Greenan, Kevin M et al, “Flat XOR-based erasure codes in storage systems”, http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf

[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode

[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version

[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, “NTT SHSS Erasure Coding backend”

[6] Jim Cheung <support@phazr.io>, “Phazr.IO libphazr erasure code backend with built-in privacy”

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.6.4

1.6.3

1.6.2

This version

1.6.2rc1 pre-release

1.6.1

1.6.0

1.5.0

1.4.0

1.3.1

1.2.0

1.1.1

1.1.0

1.0.9

1.0.7

1.0.6

1.0.4

1.0.3

1.0.2

1.0.1

1.0c2 pre-release

1.0c1 pre-release

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.2

0.9.1

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.17

0.1.16

0.1.15

0.1.14

0.1.12

0.1.11

0.1.10

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyeclib-1.6.2rc1.tar.gz (51.5 kB view details)

Uploaded Source

Built Distributions

pyeclib-1.6.2rc1-cp35-abi3-musllinux_1_1_x86_64.whl (511.0 kB view details)

Uploaded CPython 3.5+ musllinux: musl 1.1+ x86-64

pyeclib-1.6.2rc1-cp35-abi3-musllinux_1_1_aarch64.whl (471.5 kB view details)

Uploaded CPython 3.5+ musllinux: musl 1.1+ ARM64

pyeclib-1.6.2rc1-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (492.2 kB view details)

Uploaded CPython 3.5+ manylinux: glibc 2.17+ ARM64

pyeclib-1.6.2rc1-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (532.5 kB view details)

Uploaded CPython 3.5+ manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

pyeclib-1.6.2rc1-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (531.5 kB view details)

Uploaded CPython 2.7mu manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

pyeclib-1.6.2rc1-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (531.6 kB view details)

Uploaded CPython 2.7m manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

File details

Details for the file pyeclib-1.6.2rc1.tar.gz.

File metadata

  • Download URL: pyeclib-1.6.2rc1.tar.gz
  • Upload date:
  • Size: 51.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for pyeclib-1.6.2rc1.tar.gz
Algorithm Hash digest
SHA256 dd56d54d51acb70203ce43b20fb19bbf4a8ff95c0893e40b13ab120e9c207574
MD5 698174bd6b071537557bbbdf798b6c70
BLAKE2b-256 6580aef04c999f4c31f1bad514c1f11e3e65ebe2c2d962f26060a06f22f66cc4

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2rc1-cp35-abi3-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2rc1-cp35-abi3-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 06499db1cb373ff388247ecfd1f2cb0445ff1cca4e117b300d6c91f2c5745480
MD5 f44b48bcdbc0a9abfe454cb148548040
BLAKE2b-256 d3fac808f3b2bd1f14a617242ed95aa41264e86f53fb1dd9b6ecef77b67edfb8

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2rc1-cp35-abi3-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2rc1-cp35-abi3-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 8c5cdf93a26a9d1588ff9c10cb2ba44d614d7021fd150e61aa45f361a8dc3ff1
MD5 9729d15f13c3e1484276fa6d6db2be7a
BLAKE2b-256 269f136eb0c369ca18e0fe81cdff27aa4f02aedb0a719112b703f476f3dc5dbd

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2rc1-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2rc1-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 36329c59b4425d065c42d4a106b396bce210d3f77020168616a99e1bda509c0f
MD5 9e0179e55e9a56edd19bfbca8ea0e9eb
BLAKE2b-256 c31d232b500ae26ea0cb39c1ac004acdffe01415f48d399ba6f4bec42f146adf

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2rc1-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2rc1-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b66739e6488af54d8d031e3bd93a4cfabe24ae242e92a276f6e1d7f91df04982
MD5 ff78a728482ba4ad8848010dff2e0e39
BLAKE2b-256 5201f08758f55dd0952d1140d8e8e53ac382c40cd3cf0f2aa5d6b634a85298d0

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2rc1-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2rc1-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b8941bf37d1d57475f4bd78551b9657c9872f2c4504df751b185398e3168f949
MD5 0c9f85c37d4a68ea8a958ce568bc2637
BLAKE2b-256 369f61d00c6ad4b0f7c6dc03d50f3ca9f4fe9fc2825fa5691c5704af62d18c3b

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2rc1-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2rc1-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 8d2485537e528c69ee9b3bd15b28091000f6c353473e0731644c90450b920cad
MD5 4c67ac77ca37096ccb9d1e75e5751a5d
BLAKE2b-256 9433b55f3bfea702b43eaeed962acbbf62b000910ea024493beb497fde2fe1d7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page