rsonlite 0.1.0

Abstract

rsonlite is the easiest way to create custom indented data file formats in Python. It is a single small module that can be shipped with your application, or it can be easy-installed or pipped from PyPI.

Background

The rson project contains a parser that allows experimentation with variations on readable file formats, and defines a base RSON profile that is a strict superset of JSON (properly subclassed, the RSON parser passes the Python simplejson testsuite, which itself creates a superset of JSON).

RSON was born out of frustration with the size and speed of the available pure-Python YAML parsers at the time, and was designed with very similar goals to YAML – key to these formats is the use of indentation as syntax, removal of unnecessary syntactical elements such as quotes whenever possible, the ability to add comments, and the ability to use JSON as valid input.

The base RSON format definition was designed to be easily parseable from any language, but the Python parser itself is both flexible and fast, allowing for variant formats to be easily designed and tested. One such format is something that easily round-trips to XML and another format allows macros and include processing.

But the flexibility comes at a price. There is more code, and the learning curve for all the options is more steep than it would be if the parser was not that flexible, and the documentation – well, it would take a long time to do a decent job on properly documenting all the options, so that hasn’t happened yet.

rsonlite is a pared-down version of rson that primarily handles the semantics of the indentation, allowing client code to do higher-level parsing. Unlike JSON or base RSON, which distinguish between ‘true’, ‘false’, ‘null’, strings, and numbers, or simplejson, which adds ‘Nan’ and ‘Infinity’ to the mix, rsonlite divides the world up into container nodes and leaf nodes, and every leaf node is a string.

Introduction

rsonlite is a Python library that makes it easy to build a file parser for declarative hierarchical data structures using indentation. (Spaces only – tabs are not considered indentation.)

Examples

I have shamelessly stolen most of these examples from the JSON example page, because JSON is an excellent thing to compare and contrast rsonlite against.

Example 1:

>>> import rsonlite
>>> jsonstr1 = '''
... {
...     "glossary": {
...         "title": "example glossary",
...         "GlossDiv": {
...             "title": "S",
...             "GlossList": {
...                 "GlossEntry": {
...                     "ID": "SGML",
...                     "SortAs": "SGML",
...                     "GlossTerm": "Standard Generalized Markup Language",
...                     "Acronym": "SGML",
...                     "Abbrev": "ISO 8879:1986",
...                     "GlossDef": {
...                         "para": "A meta-markup language",
...                         "GlossSeeAlso": ["GML", "XML"]
...                     },
...                     "GlossSee": "markup"
...                 }
...             }
...         }
...     }
... }
... '''
>>>
>>> rsonstr1 = '''
...     glossary
...         title = example glossary
...         GlossDiv
...             title = S
...             GlossList
...                 GlossEntry
...                     ID = SGML
...                     SortAs = SGML
...                     GlossTerm = Standard Generalized Markup Language
...                     Acronym = SGML
...                     Abbrev = ISO 8879:1986
...                     GlossDef
...                         para = A meta-markup language
...                         GlossSeeAlso = [GML, XML]
...                     GlossSee = markup
... '''
>>>
>>> jsondata1 = eval(jsonstr1)
>>> rsondata1 = rsonlite.simpleparse(rsonstr1)
>>> jsondata1 == rsondata1
True

Example 2:

>>> jsonstr2 = '''
... {"menu": {
...     "id": "file",
...     "value": "File",
...     "popup": {
...         "menuitem": [
...         {"value": "New", "onclick": "CreateNewDoc()"},
...         {"value": "Open", "onclick": "OpenDoc()"},
...         {"value": "Close", "onclick": "CloseDoc()"}
...         ]
...     }
... }}
... '''
>>>
>>> rsonstr2 = '''
...     menu
...         id = file
...         value = File
...         popup
...             menuitem
...                 value = New
...                 onclick = CreateNewDoc()
...                 value = Open
...                 onclick = OpenDoc()
...                 value = Close
...                 onclick = CloseDoc()
... '''
>>>
>>> jsondata2 = eval(jsonstr2)
>>> rsondata2 = rsonlite.simpleparse(rsonstr2)
>>> jsondata2 == rsondata2
True

API

>>> rsondata1 = rsonlite.loads(rsonstr1)
>>> roundtrip = rsonlite.dumps(rsondata1, initial_indent='    ')
>>> roundtrip == rsonstr1[1:]    # Get past initial \n
True

>>> rsondata2 = rsonlite.loads(rsonstr2)
>>> roundtrip = rsonlite.dumps(rsondata2, initial_indent='    ')
>>> roundtrip == rsonstr2[1:]    # Get past initial \n
True

>>> data = rsonlite.loads(rsonstr2)
>>> pretty = rsonlite.pretty(data)
>>> print pretty,
[
    ('menu', [
        ('id', ['file']),
        ('value', ['File']),
        ('popup', [
            ('menuitem', [
                ('value', ['New']),
                ('onclick', ['CreateNewDoc()']),
                ('value', ['Open']),
                ('onclick', ['OpenDoc()']),
                ('value', ['Close']),
                ('onclick', ['CloseDoc()']),
            ])
        ])
    ])
]
>>> eval(pretty) == data
True
>>>

Tests

Empty input:

>>> rsonlite.loads('') == rsonlite.loads('   \n \n     \n') == []
True

Bad indentation:

>>> rsonlite.loads('a\n    b\n  c')
Traceback (most recent call last):
  File "<doctest README.txt[22]>", line 1, in <module>
    rsonlite.loads('a\n    b\n  c')
  File "/home/pat/projects/opensource/rson/lite/rsonlite.py", line 179, in loads
    raise err
IndentationError: unindent does not match any outer indentation level (<rsonlite>, line 3)

Multiline data, with special characters:

>>> teststr = '''
...     My key =
...            This is indented #=
...       This should be at the left edge ===#
...           This is indented differently
... '''
>>> test2 = '\n'.join(x[6:] for x in teststr.splitlines()[2:])
>>> rsonlite.loads(teststr) == [('My key', [test2])]
True

>>> teststr = '''
...     My key =  Something on the first line is at the left edge
...            This is indented
...       This should be at the left edge
...           This is indented differently
... '''
>>> test2 = teststr.split('=', 1)[1].strip().splitlines()
>>> for i in range(1, len(test2)):
...   test2[i] = test2[i][6:]
>>> test2 = '\n'.join(test2)
>>> rsonlite.loads(teststr) == [('My key', [test2])]
True

Comments and special string stuff:

>>> test1 = '''
...   a
...      b = 1
...       This is part of b's string
...      c
...   d = 2
... '''
>>> test2 = '''
... # Comments can start anywhere outside a string
...   a   # Like here  (but not below)
...      b = 1
...       This is part of b's string
...      # This isn't in the string because it's far enough left
...      # Any non-key string can start with '='
...      = c
...      # This isn't in the string because it's far enough left
...   d = 2
... '''
>>> rsonlite.loads(test1) == rsonlite.loads(test2)
True