Skip to main content

Data transformation framework for LinkML data models

Project description

Koza - a data transformation framework

Pyversions PyPi Github Action

pupa

Documentation

Disclaimer: Koza is in beta - we are looking for testers!

Overview

  • Transform csv, json, yaml, jsonl, and xml and converting them to a target csv, json, or jsonl format based on your dataclass model.
  • Koza also can output data in the KGX format
  • Write data transforms in semi-declarative Python
  • Configure source files, expected columns/json properties and path filters, field filters, and metadata in yaml
  • Create or import mapping files to be used in ingests (eg id mapping, type mappings)
  • Create and use translation tables to map between source and target vocabularies

Installation

Koza is available on PyPi and can be installed via pip/pipx:

[pip|pipx] install koza

Usage

NOTE: As of version 0.2.0, there is a new method for getting your ingest's KozaApp instance. Please see the updated documentation for details.

See the Koza documentation for usage information

Try the Examples

Validate

Give Koza a local or remote csv file, and get some basic information (headers, number of rows)

koza validate \
  --file https://raw.githubusercontent.com/monarch-initiative/koza/main/examples/data/string.tsv \
  --delimiter ' '

Sending a json or jsonl formatted file will confirm if the file is valid json or jsonl

koza validate \
  --file ./examples/data/ZFIN_PHENOTYPE_0.jsonl.gz \
  --format jsonl
koza validate \
  --file ./examples/data/ddpheno.json.gz \
  --format json

Transform

Run the example ingest, "string/protein-links-detailed"

koza transform \
  --source examples/string/protein-links-detailed.yaml \
  --global-table examples/translation_table.yaml

koza transform \
  --source examples/string-declarative/protein-links-detailed.yaml \
  --global-table examples/translation_table.yaml

Note: Koza expects a directory structure as described in the above example
with the source config file and transform code in the same directory:

.
├── ...
│   ├── your_source
│   │   ├── your_ingest.yaml
│   │   └── your_ingest.py
│   └── some_translation_table.yaml
└── ...

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

koza-0.7.2.tar.gz (28.2 kB view details)

Uploaded Source

Built Distribution

koza-0.7.2-py3-none-any.whl (36.3 kB view details)

Uploaded Python 3

File details

Details for the file koza-0.7.2.tar.gz.

File metadata

  • Download URL: koza-0.7.2.tar.gz
  • Upload date:
  • Size: 28.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.15 Linux/6.8.0-1014-azure

File hashes

Hashes for koza-0.7.2.tar.gz
Algorithm Hash digest
SHA256 1405e5a6ac97f9010167cfa45c9d832d67172fd317dc879c459ad2813392a1f9
MD5 4d5df62e8e9dfb16d1d24592be5e956f
BLAKE2b-256 2671cdc673e1235fcff5035a1289a0dbb1f37daeb73940e111204e1311a42fef

See more details on using hashes here.

File details

Details for the file koza-0.7.2-py3-none-any.whl.

File metadata

  • Download URL: koza-0.7.2-py3-none-any.whl
  • Upload date:
  • Size: 36.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.15 Linux/6.8.0-1014-azure

File hashes

Hashes for koza-0.7.2-py3-none-any.whl
Algorithm Hash digest
SHA256 2fb1fd5c5457b4e725536f05c7bc342d026dbd2af4971c833dfd77e44ffd0e18
MD5 92c6abdbfac908fe7b84c9309b82d6ff
BLAKE2b-256 50d6bdf8c2a9088d68b2b18fd63ca01bc81d834ea01feb64c9100c49b3adb226

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page