Skip to main content

compact, efficient, extensible binary serialization format

Project description

extprot: compact, efficient, extensible binary serialization format

This is a python implementation of the 'extprot' serialization scheme, the
details of which are described at:

http://eigenclass.org/R2/writings/extprot-extensible-protocols-intro

Similar to Google's Protocol Buffers and Apache Thrift, extprot allows the
definition of structured data "messages". Messages are essentially a set
of typed key-value pairs that can be efficiently serialized to/from a
compact binary format, and are defined in a language-neutral "protocol" file.
Here's a simple example of an extprot message:

message person = {
id: int;
name: string;
emails: [ string ]
}

Here the 'person' message contains three fields: 'id' is an integer, 'name'
is a string, and 'emails' is a list of strings. Such protocol descriptions
are compiled into a set of Python classes that can be manipulated using
standard syntax and idioms. If the above protocol is recorded in the file
"person.proto", here's a simple example of how it might be used:

>>> extprot.import_protocol("person.proto",globals())
>>> p1 = person(1,"Guido")
>>> print p1.emails # fields use a sensible default if possible
[]
>>> p1.emails.append("guido@python.org")
>>> p1.emails.append(7) # all fields are dynamically typechecked
Traceback (mosts recent call last):
...
ValueError: not a valid String: 7
>>> print repr(p1.to_string())
'\x01\x1f\x03\x00\x02\x03\x05Guido\x05\x13\x01\x03\x10guido@python.org'
>>> print person.from_string(p1.to_string()).name
'Guido'
>>>

Extprot compares favourably to related serialization technologies:

* powerful type system; strongly-typed tuples and lists, tagged disjoint
unions, parametric polymorphism.
* self-delimiting data; all serialized messages indicate their length,
allowing easy streaming and skipping of messages.
* self-describing data; the 'skeleton' of a message can be reconstructed
without having the protocol definition.
* compact binary format; comparable to protocol-buffers/thrift, but with
some overhead due to self-delimiting nature.

These features combine to make extprot strongly extensible, often allowing
messages to maintain backward *and* forward compatibility across protocol
extensions that include: adding fields to a message, adding elements to a
tuple, adding cases to a disjoint union, and promoting a primitive type into
a tuple, list or union.

The function extprot.import_protocol() will dynamically load a protocol file
and convert it into the corresponding python class structure. This is quite
convenient while developing a protocol since it avoids an extra compilation
step, but it does add some startup overhead and requires the pyparsing module.

To compile a protocol definition into python sourcecode for the corresponding
class definitions, use the function extprot.compile_protocol() or pipe the file
through extprot/compiler.py like so:

$ cat mydefs.proto | python extprot/compiler.py > mydefs.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

extprot-0.2.4.tar.gz (23.7 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page