Newline delimited JSON I/O that is hot swappable with csv.DictReader/Writer
Project description
NewlineJSON
===========
Currently under development.
Streaming newline delimited JSON I/O
[![Build Status](https://travis-ci.org/geowurster/NewlineJSON.svg)](https://travis-ci.org/geowurster/NewlineJSON)
Overview
--------
Read and write files with a single JSON object on every line. See the
`sample-data` directory for valid input examples.
One dictionary per line:
>>> import newlinejson
>>> with open('sample-data/dictionaries.json') as f:
>>> for line in newlinejson.Reader(f):
>>> print(line)
>>>
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'}
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'}
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'}
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'}
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
One list per line:
>>> import newlinejson
>>> with open('sample-data/lists-no-header.json') as f:
>>> for line in newlinejson.Reader(f):
>>> print(line)
>>>
['l1f2', 'l1f3', 'l1f1']
['l2f2', 'l3f3', 'l2f1']
['l3f2', 'l3f3', 'l3f1']
['l4f2', 'l4f3', 'l4f1']
['l5f2', 'l5f3', 'l5f1']
Mixed content:
>>> import newlinejson
>>> with open('sample-data/mixed-content.json') as f:
>>> for line in newlinejson.Reader(f):
>>> print(line)
>>>
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'}
['l1f2', 'l1f3', 'l1f1']
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'}
['l2f2', 'l3f3', 'l2f1']
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'}
['l3f2', 'l3f3', 'l3f1']
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'}
['l4f2', 'l4f3', 'l4f1']
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
['l5f2', 'l5f3', 'l5f1']
The standard JSON functions `load/s()` and `dump/s()` are still available but
should ONLY be used on small files and are really only included as a convenience.
The `load/s()` functions return lists of JSON objects and `dump/s()`take the
same format.
Load from a file:
>>> import newlinejson
>>> with open('sample-data/dictionaries.json') as f:
>>> print(newlinejson.load(f))
>>>
[
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'},
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'},
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'},
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'},
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
]
Load from a string:
>>> import newlinejson
>>> with open('sample-data/dictionaries.json') as f:
>>> print(newlinejson.loads(f.read()))
>>>
[
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'},
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'},
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'},
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'},
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
]
Dump to a file:
>>> with open('output.json', 'w') as f:
>>> newlinejson.dump(json_lines, f)
Dump to a string:
>>> string = newlinejson.dumps(json_lines)
Installing
----------
$ pip install newlinejson
Developing
----------
$ pip install virtualenv
$ git clone https://github.com/geowurster/NewlineJSON
$ cd NewlineJSON
$ virtualenv venv
$ source venv/bin/activate
$ pip install -r requirements.txt -r requirements-dev.txt
$ nosetests
Testing
-------
Code coverage report
$ nosetests \
$ --with-coverage \
$ --cover-package=newlinejson \
$ --cover-erase --cover-inclusive
PEP8 report - the default style guide is used except a max line length of 120
is preferred.
$ pep8 --max-line-length=120 newlinejson
===========
Currently under development.
Streaming newline delimited JSON I/O
[![Build Status](https://travis-ci.org/geowurster/NewlineJSON.svg)](https://travis-ci.org/geowurster/NewlineJSON)
Overview
--------
Read and write files with a single JSON object on every line. See the
`sample-data` directory for valid input examples.
One dictionary per line:
>>> import newlinejson
>>> with open('sample-data/dictionaries.json') as f:
>>> for line in newlinejson.Reader(f):
>>> print(line)
>>>
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'}
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'}
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'}
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'}
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
One list per line:
>>> import newlinejson
>>> with open('sample-data/lists-no-header.json') as f:
>>> for line in newlinejson.Reader(f):
>>> print(line)
>>>
['l1f2', 'l1f3', 'l1f1']
['l2f2', 'l3f3', 'l2f1']
['l3f2', 'l3f3', 'l3f1']
['l4f2', 'l4f3', 'l4f1']
['l5f2', 'l5f3', 'l5f1']
Mixed content:
>>> import newlinejson
>>> with open('sample-data/mixed-content.json') as f:
>>> for line in newlinejson.Reader(f):
>>> print(line)
>>>
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'}
['l1f2', 'l1f3', 'l1f1']
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'}
['l2f2', 'l3f3', 'l2f1']
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'}
['l3f2', 'l3f3', 'l3f1']
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'}
['l4f2', 'l4f3', 'l4f1']
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
['l5f2', 'l5f3', 'l5f1']
The standard JSON functions `load/s()` and `dump/s()` are still available but
should ONLY be used on small files and are really only included as a convenience.
The `load/s()` functions return lists of JSON objects and `dump/s()`take the
same format.
Load from a file:
>>> import newlinejson
>>> with open('sample-data/dictionaries.json') as f:
>>> print(newlinejson.load(f))
>>>
[
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'},
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'},
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'},
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'},
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
]
Load from a string:
>>> import newlinejson
>>> with open('sample-data/dictionaries.json') as f:
>>> print(newlinejson.loads(f.read()))
>>>
[
{'field2': 'l1f2', 'field3': 'l1f3', 'field1': 'l1f1'},
{'field2': 'l2f2', 'field3': 'l3f3', 'field1': 'l2f1'},
{'field2': 'l3f2', 'field3': 'l3f3', 'field1': 'l3f1'},
{'field2': 'l4f2', 'field3': 'l4f3', 'field1': 'l4f1'},
{'field2': 'l5f2', 'field3': 'l5f3', 'field1': 'l5f1'}
]
Dump to a file:
>>> with open('output.json', 'w') as f:
>>> newlinejson.dump(json_lines, f)
Dump to a string:
>>> string = newlinejson.dumps(json_lines)
Installing
----------
$ pip install newlinejson
Developing
----------
$ pip install virtualenv
$ git clone https://github.com/geowurster/NewlineJSON
$ cd NewlineJSON
$ virtualenv venv
$ source venv/bin/activate
$ pip install -r requirements.txt -r requirements-dev.txt
$ nosetests
Testing
-------
Code coverage report
$ nosetests \
$ --with-coverage \
$ --cover-package=newlinejson \
$ --cover-erase --cover-inclusive
PEP8 report - the default style guide is used except a max line length of 120
is preferred.
$ pep8 --max-line-length=120 newlinejson
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
NewlineJSON-0.1.0.tar.gz
(11.7 kB
view details)
File details
Details for the file NewlineJSON-0.1.0.tar.gz
.
File metadata
- Download URL: NewlineJSON-0.1.0.tar.gz
- Upload date:
- Size: 11.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61d881bc1017cf69a7bfc34ebaafae3ee4a2810861135d093a8dfce5184bdca7 |
|
MD5 | 39e6df5753b1234df32b15798ab2c27a |
|
BLAKE2b-256 | d7f3e6b464b76f7c37711094378f45ab500f19f415091853c6217dbd9d58b8d2 |