libraptorq 16.2.1

Command-line script

To encode file, with 50% extra symbols (resulting indivisible data chunks to be stored/transmitted intact or lost entirely) and 30% of total from these (K required symbols + X repair symbols) dropped (just for testing purposes) before saving them to “setup.py.enc”:

% ./rq --debug encode --repair-symbols-rate 0.5 --drop-rate 0.3 setup.py setup.py.enc
2016-02-25 03:21:09 :: DEBUG :: Encoded 1,724 B into 168 symbols\
  (needed: >108, repair rate: 50%), 48 dropped (30%), 120 left in output (1,920 B without ids)

Decode original file back from these:

% ./rq --debug decode setup.py.enc setup.py.dec
2016-02-25 03:21:34 :: DEBUG :: Decoded 1,721 B of data\
  from 108 processed symbols (1,728 B without ids, symbols total: 120)

% sha256sum -b setup.py{,.dec}
36c50348459b51821a2715b0f5c4ef08647d66f77a29913121af4f0f4dfef454 *setup.py
36c50348459b51821a2715b0f5c4ef08647d66f77a29913121af4f0f4dfef454 *setup.py.dec

No matter which chunks are dropped (get picked by random.choice), file should be recoverable from output as long as number of chunks left (in each “block”) is slightly (by ~0.02%) above K.

Output data (“setup.py.enc” in the example) for the script is JSON-encoded list of base64-encoded symbols, as well as some parameters for lib init (oti_scheme, oti_common) and input data length.

See output with –help option for all the other script parameters.

Python module

To use as a python2 module:

from libraptorq import RQEncoder

data = 'some input string' * 500

# Data size must be divisible by RQEncoder.data_size_div
data_len, n = len(data), RQEncoder.data_size_div
if data_len % n: data += '\0' * (n - data_len % n)

with RQEncoder(data, min_subsymbol_size=4, symbol_size=16, max_memory=200) as enc:

  symbols = dict()
  oti_scheme, oti_common = enc.oti_scheme, enc.oti_common

  for block in enc:
    symbols.update(block.encode_iter(repair_rate=0))

data_encoded = data_len, oti_scheme, oti_common, symbols

oti_scheme and oti_common are two integers specifying encoder options, needed to initialize decoder, which can be hard-coded (if constant) on both ends.

Data length must be divisible by

block.encode_iter() can be used without options to produce max possible amount of symbols, up to block.symbols + block.max_repair. Above example only produces K symbols - min amount required.

For decoding (reverse operation):

from libraptorq import RQDecoder

data_len, oti_scheme, oti_common, symbols = data_encoded

with RQDecoder(oti_scheme, oti_common) as dec:
  for sym_id, sym in symbols.viewitems(): dec.add_symbol(sym, sym_id)

  data = dec.decode()[:data_len]

Note that in practice, e.g. when transmitting each symbol in a udp packet, one’d want to send something like sym_id || sym_data || checksum, and keep sending these from block.encode_iter() until other side acknowledges that it can decode a block (i.e. enough symbols received, see RQDecoder.decode_block()), then start streaming the next block in similar fashion.

See __main__.py file (cli script) for an extended example, and libRaptorQ docs for info on its API, which this module wraps around.