Data transformation framework for LinkML data models
Project description
Koza - a data transformation framework
Disclaimer: Koza is in beta - we are looking for testers!
Overview
- Transform csv, json, yaml, jsonl, and xml and converting them to a target csv, json, or jsonl format based on your dataclass model.
- Koza also can output data in the KGX format
- Write data transforms in semi-declarative Python
- Configure source files, expected columns/json properties and path filters, field filters, and metadata in yaml
- Create or import mapping files to be used in ingests (eg id mapping, type mappings)
- Create and use translation tables to map between source and target vocabularies
Installation
Koza is available on PyPi and can be installed via pip/pipx:
[pip|pipx] install koza
Usage
NOTE: As of version 0.2.0, there is a new method for getting your ingest's KozaApp instance. Please see the updated documentation for details.
See the Koza documentation for usage information
Try the Examples
Validate
Give Koza a local or remote csv file, and get some basic information (headers, number of rows)
koza validate \
--file https://raw.githubusercontent.com/monarch-initiative/koza/main/examples/data/string.tsv \
--delimiter ' '
Sending a json or jsonl formatted file will confirm if the file is valid json or jsonl
koza validate \
--file ./examples/data/ZFIN_PHENOTYPE_0.jsonl.gz \
--format jsonl
koza validate \
--file ./examples/data/ddpheno.json.gz \
--format json \
--compression gzip
Transform
Run the example ingest, "string/protein-links-detailed"
koza transform \
--source examples/string/protein-links-detailed.yaml \
--global-table examples/translation_table.yaml
koza transform \
--source examples/string-declarative/protein-links-detailed.yaml \
--global-table examples/translation_table.yaml
Note:
Koza expects a directory structure as described in the above example
with the source config file and transform code in the same directory:
.
├── ...
│ ├── your_source
│ │ ├── your_ingest.yaml
│ │ └── your_ingest.py
│ └── some_translation_table.yaml
└── ...
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file koza-0.5.0.tar.gz.
File metadata
- Download URL: koza-0.5.0.tar.gz
- Upload date:
- Size: 76.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.7.0 CPython/3.10.13 Linux/6.2.0-1015-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7dda93c872362da1007827be396201e98e4b9c55c1ec62936c2a258d46801409 |
|
| MD5 | 7f4a7a53b76907c14e7c05fdd069dafd |
|
| BLAKE2b-256 | c27d9a7afbd49d8bfaf8b923938f25e95e9a397ce4e684515c323318c6a6e351 |
File details
Details for the file koza-0.5.0-py3-none-any.whl.
File metadata
- Download URL: koza-0.5.0-py3-none-any.whl
- Upload date:
- Size: 85.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.7.0 CPython/3.10.13 Linux/6.2.0-1015-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 859e53b53c6b692212763bdfb80bd6bc363098e9f49c4aa78a6b23d421740867 |
|
| MD5 | fcdaa5ed414874f3cb45a3ff5bf47fb8 |
|
| BLAKE2b-256 | 988fc18f054f985d30d96c93df70fbaa94462e067dd2d75fdd39ad706a26772c |
