Skip to main content

A Python library and set of command line utilities for exchanging Knowledge Graphs (KGs) that conform to or are aligned to the Biolink Model.

Project description

Knowledge Graph Exchange

Python Run testsDocumentation Status Quality Gate Status Maintainability Rating Coverage PyPI Docker

KGX (Knowledge Graph Exchange) is a Python library and set of command line utilities for exchanging Knowledge Graphs (KGs) that conform to or are aligned to the Biolink Model.

The core datamodel is a Property Graph (PG), represented internally in Python using a networkx MultiDiGraph model.

KGX allows conversion to and from:

KGX will also provide validation, to ensure the KGs are conformant to the Biolink Model: making sure nodes are categorized using Biolink classes, edges are labeled using valid Biolink relationship types, and valid properties are used.

Internal representation is a property graph, specifically a networkx MultiDiGraph.

The structure of this graph is expected to conform to the Biolink Model standard, as specified in the KGX format specification.

In addition to the main code-base, KGX also provides a series of command line operations.

Error Detection and Reporting

Non-redundant JSON-formatted structured error logging is now provided in KGX Transformer, Validator, GraphSummary and MetaKnowledgeGraph operations. See the various unit tests for the general design pattern (using the Validator as an example here):

from kgx.validator import Validator
from kgx.transformer import Transformer

Validator.set_biolink_model("2.11.0")

# Validator assumes the currently set Biolink Release
validator = Validator()

transformer = Transformer(stream=True)

transformer.transform(
    input_args = {
        "filename": [
            "graph_nodes.tsv",
            "graph_edges.tsv",
        ],
        "format": "tsv",
    },
    output_args={
        "format": "null"
    },
    inspector=validator,
)

# Both the Validator and the Transformer can independently capture errors

# The Validator, from the overall semantics of the graph...
# Here, we just report severe Errors from the Validator (no Warnings)
validator.write_report(open("validation_errors.json", "w"), "Error")

# The Transformer, from the syntax of the input files... 
# Here, we catch *all* Errors and Warnings (by not providing a filter)
transformer.write_report(open("input_errors.json", "w"))

The JSON error outputs will look something like this:

{
    "ERROR": {
        "MISSING_EDGE_PROPERTY": {
            "Required edge property 'id' is missing": [
                "A:123->X:1",
                "B:456->Y:2"
            ],
            "Required edge property 'object' is missing": [
                "A:123->X:1"
            ],
            "Required edge property 'predicate' is missing": [
                "A:123->X:1"
            ],
            "Required edge property 'subject' is missing": [
                "A:123->X:1",
                "B:456->Y:2"
            ]
        }
    },
    "WARNING": {
        "DUPLICATE_NODE": {
          "Node 'id' duplicated in input data": [
            "MONDO:0010011",
            "REACT:R-HSA-5635838"
          ]
        }
    }
}

This system reduces the significant redundancies of earlier line-oriented KGX logging text output files, in that graph entities with the same class of error are simply aggregated in lists of names/identifiers at the leaf level of the JSON structure.

The top level JSON tags originate from the MessageLevel class and the second level tags from the ErrorType class in the error_detection module, while the third level messages are hard coded as log_error method messages in the code.

It is likely that additional error conditions within KGX can be efficiently captured and reported in the future using this general framework.

Installation

The installation for KGX requires Python 3.9 or greater.

Installation for users

Installing from PyPI

KGX is available on PyPI and can be installed using pip as follows,

pip install kgx

To install a particular version of KGX, be sure to specify the version number,

pip install kgx==0.5.0

Installing from GitHub

Clone the GitHub repository and then install,

git clone https://github.com/biolink/kgx
cd kgx
python setup.py install

Installation for developers

Setting up a development environment

To build directly from source, first clone the GitHub repository,

git clone https://github.com/biolink/kgx
cd kgx

Then install the necessary dependencies listed in requirements.txt,

pip3 install -r requirements.txt

For convenience, make use of the venv module in Python3 to create a lightweight virtual environment,

python3 -m venv env
source env/bin/activate

pip install -r requirements.txt

To install KGX you can do one of the following,

pip install .

# OR 

python setup.py install

Setting up a testing environment for Neo4j

This release of KGX supports graph source and sink transactions with the 4.3 release of Neo4j.

KGX has a suite of tests that rely on Docker containers to run Neo4j specific tests.

To set up the required containers, first install Docker on your local machine.

Once Docker is up and running, run the following commands:

docker run -d --rm --name kgx-neo4j-integration-test \
            -p 7474:7474 -p 7687:7687 \
            --env NEO4J_AUTH=neo4j/test  \
            neo4j:4.3
docker run -d --rm --name kgx-neo4j-unit-test  \
            -p 8484:7474 -p 8888:7687 \
            --env NEO4J_AUTH=neo4j/test \
            neo4j:4.3

Note: Setting up the Neo4j container is optional. If there is no container set up then the tests that rely on them are skipped.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kgx-1.6.2.tar.gz (95.3 kB view details)

Uploaded Source

Built Distributions

kgx-1.6.2-py3.9.egg (112.6 kB view details)

Uploaded Source

kgx-1.6.2-py3-none-any.whl (115.4 kB view details)

Uploaded Python 3

File details

Details for the file kgx-1.6.2.tar.gz.

File metadata

  • Download URL: kgx-1.6.2.tar.gz
  • Upload date:
  • Size: 95.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.7

File hashes

Hashes for kgx-1.6.2.tar.gz
Algorithm Hash digest
SHA256 6b98af0f5ee73aa80b1e0a6ae00bfcf1749d844164c9ea416e3d0cf2082ce87a
MD5 7e48fb1e54bf2d8d21a0b22734f7c3c2
BLAKE2b-256 df9907caa8e9fb3d294bb3b2ac671485e64199b6963e9afe829e5bfee6427532

See more details on using hashes here.

Provenance

File details

Details for the file kgx-1.6.2-py3.9.egg.

File metadata

  • Download URL: kgx-1.6.2-py3.9.egg
  • Upload date:
  • Size: 112.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.7

File hashes

Hashes for kgx-1.6.2-py3.9.egg
Algorithm Hash digest
SHA256 e316e69cdc6108c22860c3f1034313fe89111e9e3cd1b7bc8460f8a6d1de2556
MD5 8eb6448344160222a607f416d66ad1ec
BLAKE2b-256 91f75b317687309be712a65e34f0623ddb84079eba2f5cd13efcbd555e391e41

See more details on using hashes here.

Provenance

File details

Details for the file kgx-1.6.2-py3-none-any.whl.

File metadata

  • Download URL: kgx-1.6.2-py3-none-any.whl
  • Upload date:
  • Size: 115.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.7

File hashes

Hashes for kgx-1.6.2-py3-none-any.whl
Algorithm Hash digest
SHA256 a9df23178b30fc82764577661e6cfb799c1a254f93dee1c6a8cd00d3612e9e4f
MD5 1002eb077ec95411b4a38e9d9b3465fa
BLAKE2b-256 c797b0c36d08ca096c3e456071ad696b3dbbbee0847dcb83565ea1223db08f93

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page