This library provides a simple Python interface for implementing erasure codes. To obtain the best possible performance, the underlying erasure code algorithms are written in C.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

This library provides a simple Python interface for implementing erasure codes and is known to work with Python 2.7 and 3.5 through 3.13. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.

PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3], Intel’s ISA-L [4] and Phazr.IO’s libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010[2]). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).

Installation

Install pre-requisites:

  • Python 2.7 or 3.5+ (including development packages), argparse, setuptools

  • liberasurecode v1.4.0 or greater [3]

  • Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4], etc

Install dependencies:

Debian/Ubuntu hosts:

$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirements.txt

RHEL/CentOS hosts:

$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
$ sudo pip install -U bindep -r test-requirements.txt
$ tools/test-setup.sh

If you want to confirm all dependency packages installed successfully, try:

$ sudo bindep -f bindep.txt

For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.

Install PyECLib:

$ sudo python setup.py install

Run test suite included:

$ ./.unittests

If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to:

/etc/ld.so.conf

and then make sure to run:

$ sudo ldconfig

Getting started

Examples of using PyECLib are provided in the “tools” directory:

Command-line encoder:

tools/pyeclib_encode.py

Command-line decoder:

tools/pyeclib_decode.py

Utility to determine what is needed to reconstruct missing fragments:

tools/pyeclib_fragments_needed.py

A configuration utility to help compare available EC schemes in terms of performance and redundancy:

tools/pyeclib_conf_tool.py

PyEClib initialization:

ec_driver = ECDriver(k=<num_encoded_data_fragments>,
                     m=<num_encoded_parity_fragments>,
                     ec_type=<ec_scheme>))

Supported ec_type values:

  • liberasurecode_rs_vand => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]

  • jerasure_rs_vand => Vandermonde Reed-Solomon encoding, based on Jerasure [1]

  • jerasure_rs_cauchy => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]

  • flat_xor_hd_3, flat_xor_hd_4 => Flat-XOR based HD combination codes, liberasurecode [3]

  • isa_l_rs_vand => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]

  • isa_l_rs_cauchy => Cauchy Reed-Solomon encoding (ISA-L variant) [4]

  • shss => NTT Lab Japan’s Erasure Coding Library [5]

  • libphazr => Phazr.IO’s erasure code library with built-in privacy [6]

Code Maintenance

This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.

References

[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org

[2] Greenan, Kevin M et al, “Flat XOR-based erasure codes in storage systems”, http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf

[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode

[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version

[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, “NTT SHSS Erasure Coding backend”

[6] Jim Cheung <support@phazr.io>, “Phazr.IO libphazr erasure code backend with built-in privacy”

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.6.4

1.6.3

This version

1.6.2

1.6.2rc1 pre-release

1.6.1

1.6.0

1.5.0

1.4.0

1.3.1

1.2.0

1.1.1

1.1.0

1.0.9

1.0.7

1.0.6

1.0.4

1.0.3

1.0.2

1.0.1

1.0c2 pre-release

1.0c1 pre-release

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.2

0.9.1

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

0.1.17

0.1.16

0.1.15

0.1.14

0.1.12

0.1.11

0.1.10

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyeclib-1.6.2.tar.gz (50.6 kB view details)

Uploaded Source

Built Distributions

pyeclib-1.6.2-cp35-abi3-musllinux_1_1_x86_64.whl (510.9 kB view details)

Uploaded CPython 3.5+ musllinux: musl 1.1+ x86-64

pyeclib-1.6.2-cp35-abi3-musllinux_1_1_aarch64.whl (471.4 kB view details)

Uploaded CPython 3.5+ musllinux: musl 1.1+ ARM64

pyeclib-1.6.2-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (492.2 kB view details)

Uploaded CPython 3.5+ manylinux: glibc 2.17+ ARM64

pyeclib-1.6.2-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (532.4 kB view details)

Uploaded CPython 3.5+ manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

pyeclib-1.6.2-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (531.5 kB view details)

Uploaded CPython 2.7mu manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

pyeclib-1.6.2-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (531.5 kB view details)

Uploaded CPython 2.7m manylinux: glibc 2.12+ x86-64 manylinux: glibc 2.5+ x86-64

File details

Details for the file pyeclib-1.6.2.tar.gz.

File metadata

  • Download URL: pyeclib-1.6.2.tar.gz
  • Upload date:
  • Size: 50.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for pyeclib-1.6.2.tar.gz
Algorithm Hash digest
SHA256 5a6fc686bf2ede13a81d26ced4f4637ef521642d6cd00b4cc39f8bb4e80df663
MD5 7d2fdf734c21bd8186b8a42b253b735c
BLAKE2b-256 c9097626fc3d873e1028e9109a4fccc3e4306ae1a6ec3a6477278bc3ef24fb2d

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2-cp35-abi3-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2-cp35-abi3-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 2eb8295cc55618dfcc920f28c2b5cd1a017b2ffad5f6aaf2b34962f9be84d9ee
MD5 b277d5870b7548728cd28c8b6a842cb1
BLAKE2b-256 9e9db938af343548711caa14f58310a20b474cb4c01ddf3057996fd0e6223a89

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2-cp35-abi3-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2-cp35-abi3-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 41f7a2fbf6bcf8cceec7e9b58f7bb8e030c012feb14de123a37fe61aa3d17b45
MD5 8fe2bfa3becd796beb86ecac416f748d
BLAKE2b-256 e50e9f07809cae9e2cb135083a5145a953cd80d7856dda361a16a19d624be8fc

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2-cp35-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 32eb466d274762ca86dff904bdd125d4abbdbd0744bd0e6bdc7ec8026eb4e38a
MD5 500a3fb62f9741d3c68facb90b7a7d8d
BLAKE2b-256 2316970d6c4f60292fb8ebfbe7cc6644cde3464cc53c8bbad3f6aa3f77f6314d

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 6882467e728f7063bce3a256e48a4adb88c946061d20eadca010e70649fd2199
MD5 20862cd0ef94f56754eeb2f4803b1e81
BLAKE2b-256 364f5c91f06b2dd71ddbcd4e5eb5107f571236d83ec1864d60bea4f4e643273d

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 6b05b88e909b11e36c09c812e71e339564aab62ab392adf8a6b6419445d092f1
MD5 190f13f849d7e82f2a69a15249391e1d
BLAKE2b-256 f0d64c7273e81e0a5536784a03f0cbf66a1d28f83161becc77906afd644323a6

See more details on using hashes here.

File details

Details for the file pyeclib-1.6.2-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.6.2-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 0d4a4411619b547392ae43c02619b6b6bdd0550bbf561d3fbb56ca11b97ca0cb
MD5 ea8ddb5b10c66dc1dc85fadc26dbab40
BLAKE2b-256 ec9acd5c223b88d9cf97b2e9c23f4480339621afd962f1f05c6ac98ea029c195

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page