cihaidata-unihan

Tool to build UNIHAN dataset into datapackage / simple data format.

These details have not been verified by PyPI

Project links

Project description

cihaidata-unihan - tool to build unihan into simple data format CSV format. Part of the cihai project.

License

Unihan’s data is disperved across multiple files in the format of:

U+3400      kCantonese      jau1
U+3400      kDefinition     (same as U+4E18 丘) hillock or mound
U+3400      kMandarin       qiū
U+3401      kCantonese      tim2
U+3401      kDefinition     to lick; to taste, a mat, bamboo bark
U+3401      kHanyuPinyin    10019.020:tiàn
U+3401      kMandarin       tiàn

script/process.py will download Unihan.zip and build all files into a single tabular CSV (default output: ./data/unihan.csv):

char,ucn,kCantonese,kDefinition,kHanyuPinyin,kMandarin
丘,U+3400,jau1,(same as U+4E18 丘) hillock or mound,,qiū
㐁,U+3401,tim2,"to lock; to taste, a mat, bamboo bark",10019.020:"tiàn,tiàn"

process.py supports command line arguments. See script/process.py CLI arguments for information on how you can specify custom columns, files, download URL’s and output destinations.

Being built against unit tests. See the Travis Builds and Revision History.

Usage

To download and build your own unihan.csv:

$ ./scripts/process.py

Creates data/unihan.csv.

See script/process.py CLI arguments for advanced usage examples.

Structure

# dataset metadata, schema information.
datapackage.json

# (future) when this package is stable, unihan.csv will be provided
data/unihan.csv

# stores downloaded Unihan.zip and it's txt file contents (.gitignore'd)
data/build_files/

# script to download + build a SDF csv of unihan.
scripts/process.py

# unit tests to verify behavior / consistency of builder
tests/*

# python 2/3 compatibility modules
script/_compat.py
script/unicodecsv.py

# python module, public-facing python API.
__init__.py
scripts/__init__.py

# utility / helper functions
scripts/util.py

Cihai is not required for:

data/unihan.csv - simple data format compatible csv file.
scripts/process.py - create a data/unihan.csv.

When this module is stable, data/unihan.csv will have prepared releases, without requires using scripts/process.py. process.py will not require external libraries.

Examples

Related links:

CSV Simple Data Format (SDF): http://data.okfn.org/standards/simple-data-format
Tools: http://data.okfn.org/tools

Python support	Python 2.7, >= 3.3, pypy/pypy3
Source	https://github.com/cihai/cihaidata-unihan
Docs	https://cihaidata-unihan.git-pull.com
Changelog	https://cihaidata-unihan.git-pull.com/en/latest/history.html
API	https://cihaidata-unihan.git-pull.com/en/latest/api.html
Issues	https://github.com/cihai/cihaidata-unihan/issues
Travis	https://travis-ci.org/cihai/cihaidata-unihan
Test coverage	https://codecov.io/gh/cihai/cihaidata-unihan
pypi	https://pypi-hypernode.com/pypi/cihaidata-unihan
OpenHub	https://www.openhub.net/p/cihaidata-unihan
License	MIT.
git repo	`$ git clone https://github.com/cihai/cihaidata-unihan.git`
install dev	`$ git clone https://github.com/cihai/cihaidata-unihan.git cihai $ cd ./cihai $ virtualenv .env $ source .env/bin/activate $ pip install -e .`
tests	`$ python setup.py test`

Project details

These details have not been verified by PyPI

Project links

Release history Release notifications | RSS feed

0.4.2

May 8, 2017

0.4.1

May 8, 2017

This version

0.4.0

May 8, 2017

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cihaidata-unihan-0.4.0.tar.gz (11.6 kB view details)

Uploaded May 8, 2017 Source

File details

Details for the file cihaidata-unihan-0.4.0.tar.gz.

File metadata

Download URL: cihaidata-unihan-0.4.0.tar.gz
Upload date: May 8, 2017
Size: 11.6 kB
Tags: Source
Uploaded using Trusted Publishing? No

File hashes

Hashes for cihaidata-unihan-0.4.0.tar.gz
Algorithm	Hash digest
SHA256	`a551f396235bc4fed682d07de74f46865ead1793d53039f24bf15d85a98792c3`
MD5	`e24934dadf197b6e653c85798c0cb13e`
BLAKE2b-256	`99033f291c1828a33a1cae2170d0b01811e9de0ba10a131d61a8faa5575f6d63`