Tekton Compiler for Kubeflow Pipelines

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ckadner from gravatar.com ckadner Avatar for fenglixa from gravatar.com fenglixa Avatar for jinchihe from gravatar.com jinchihe Avatar for tomcli from gravatar.com tomcli Avatar for yihongwang from gravatar.com yihongwang

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache 2.0)
  • Author: kubeflow.org
  • Requires: Python >=3.5.3
Classifiers

Project description

Kubeflow Pipelines SDK for Tekton

The Kubeflow Pipelines SDK allows data scientists to define end-to-end machine learning and data pipelines. The output of the Kubeflow Pipelines SDK compiler is YAML for Argo.

The kfp-tekton SDK is extending the Compiler and the Client of the Kubeflow Pipelines SDK to generate Tekton YAML and to subsequently upload and run the pipeline with the Kubeflow Pipelines engine backed by Tekton.

Table of Contents

Extensions to the Kubeflow Pipelines SDK

In addition to the functionality provided by the Kubeflow Pipelines SDK the kfp-tekton SDK provides a TektonCompiler and a TektonClient:

TektonCompiler:

  • kfp_tekton.compiler.TektonCompiler.compile compiles Python DSL code into a YAML file containing a Tekton PipelineRun which can be deployed directly to a Tekton enabled Kubernetes cluster or uploaded to the Kubeflow Pipelines dashboard with the Tekton backend.

TektonClient:

  • kfp_tekton.TektonClient.create_run_from_pipeline_func compiles DSL pipeline function and runs the pipeline on a Kubernetes cluster with KFP and Tekton

Project Prerequisites

Follow the instructions for installing project prerequisites and take note of some important caveats.

Installation

You can install the latest release of the kfp-tekton compiler from PyPi. We recommend to create a Python virtual environment first:

python3 -m venv .venv
source .venv/bin/activate

pip install kfp-tekton

Alternatively you can install the latest version of the kfp-tekton compiler from source by cloning the repository https://github.com/kubeflow/kfp-tekton:

  1. Clone the kfp-tekton repo:

    git clone https://github.com/kubeflow/kfp-tekton.git
cd kfp-tekton

  2. Setup Python environment with Conda or a Python virtual environment:

    python3 -m venv .venv
source .venv/bin/activate

  3. Build the compiler:

    pip install -e sdk/python

  4. Run the compiler tests (optional):

    pip install pytest
make test

Compiling a Kubeflow Pipelines DSL Script

The kfp-tekton Python package comes with the dsl-compile-tekton command line executable, which should be available in your terminal shell environment after installing the kfp-tekton Python package.

If you cloned the kfp-tekton project, you can find example pipelines in the samples folder or under sdk/python/tests/compiler/testdata folder.

dsl-compile-tekton \
    --py sdk/python/tests/compiler/testdata/parallel_join.py \
    --output pipeline.yaml

Note: If the KFP DSL script contains a __main__ method calling the kfp_tekton.compiler.TektonCompiler.compile() function:

if __name__ == "__main__":
    from kfp_tekton.compiler import TektonCompiler
    TektonCompiler().compile(pipeline_func, "pipeline.yaml")

... then the pipeline can be compiled by running the DSL script with python3 executable from a command line shell, producing a Tekton YAML file pipeline.yaml in the same directory:

python3 pipeline.py

Running the Compiled Pipeline on a Tekton Cluster

After compiling the sdk/python/tests/compiler/testdata/parallel_join.py DSL script in the step above, we need to deploy the generated Tekton YAML to our Kubernetes cluster with kubectl. The Tekton server will automatically start a pipeline run. We can then follow the logs using the tkn CLI.

kubectl apply -f pipeline.yaml

tkn pipelinerun logs --last --follow

Once the Tekton Pipeline is running, the logs should start streaming:

Waiting for logs to be available...

[gcs-download : main] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate

[gcs-download-2 : main] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath

[echo : main] Text 1: With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
[echo : main]
[echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
[echo : main]

List of Available Features

To understand how each feature is implemented and its current status, please visit the FEATURES doc.

Tested Pipelines

We are testing the compiler on more than 80 pipelines found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compiler testdata folder, the KFP core samples and the samples contributed by third parties.

A report card of Kubeflow Pipelines samples that are currently supported by the kfp-tekton compiler can be found here. If you work on a PR that enables another of the missing features please ensure that your code changes are improving the number of successfully compiled KFP pipeline samples.

Troubleshooting

  • When you encounter ServiceAccount related permission issues, refer to the "Servince Account and RBAC" doc

  • If you run into the error bad interpreter: No such file or director when trying to use Python's venv, remove the current virtual environment in the .venv directory and create a new one using virtualenv .venv

  • For big data passing, PVs must be created manually, or dynamic volume provisioning, has to be enabled (https://kubernetes.io/docs/concepts/storage/dynamic-provisioning). The PVC name has to match the PipelineRun name until issue #181 has been addressed.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ckadner from gravatar.com ckadner Avatar for fenglixa from gravatar.com fenglixa Avatar for jinchihe from gravatar.com jinchihe Avatar for tomcli from gravatar.com tomcli Avatar for yihongwang from gravatar.com yihongwang

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache 2.0)
  • Author: kubeflow.org
  • Requires: Python >=3.5.3
Classifiers

Release history Release notifications | RSS feed

1.9.3

1.9.2

1.9.1

1.9.0

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.9

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.10

1.5.9

1.5.8

1.5.7

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.2

1.4.1

1.4.0

1.3.1

1.3.0

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.1

1.0.0

0.9.0

0.8.1

0.8.0

0.8.0rc0 pre-release

0.7.0

0.6.0

0.6.0rc0 pre-release

0.5.1

0.5.1rc1 pre-release

0.5.1rc0 pre-release

0.5.0

0.4.0

This version

0.3.0

0.3.0rc0 pre-release

0.2.0

0.1.0

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kfp-tekton-0.3.0.tar.gz (37.0 kB view details)

Uploaded Source

File details

Details for the file kfp-tekton-0.3.0.tar.gz.

File metadata

  • Download URL: kfp-tekton-0.3.0.tar.gz
  • Upload date:
  • Size: 37.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.1 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.5

File hashes

Hashes for kfp-tekton-0.3.0.tar.gz
Algorithm Hash digest
SHA256 368ee9c84b1f128a3302c9815663dd95c189bbc3f6f6711696e15697547bde19
MD5 9926b535125a9ef71694c4c8d290f9c6
BLAKE2b-256 196c9cf6a3674b13feae9374946f83df85027c56c3c1d9f768b8ee85f624562c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page