Jupyter Notebook operator for Kubeflow Pipelines
Project description
kfp-notebook implements Kubeflow Pipelines operator NotebookOp that supports processing of notebooks, Python scripts, and R scripts in pipelines.
Building kfp-notebook
make clean install
Usage
The example below can easily be added to a python script or jupyter notebook for testing purposes.
import os
import kfp
from kfp_notebook.pipeline import NotebookOp
from kubernetes.client.models import V1EnvVar
# KubeFlow Pipelines API Endpoint
kfp_url = 'http://dataplatform.ibm.com:32488/pipeline'
# S3 Object Storage
cos_endpoint = 'http://s3.us-south.cloud-object-storage.appdomain.cloud'
cos_bucket = 'test-bucket'
cos_username = 'test'
cos_password = 'test123'
cos_directory = 'test-directory'
cos_dependencies_archive = 'test-archive.tar.gz'
# Inputs and Outputs
inputs = []
outputs = []
# Container Image
image = 'tensorflow/tensorflow:latest'
def run_notebook_op(op_name, notebook_path):
notebook_op = NotebookOp(name=op_name,
notebook=notebook_path,
cos_endpoint=cos_endpoint,
cos_bucket=cos_bucket,
cos_directory=cos_directory,
cos_dependencies_archive=cos_dependencies_archive,
pipeline_outputs=outputs,
pipeline_inputs=inputs,
image=image)
notebook_op.container.add_env_variable(V1EnvVar(name='AWS_ACCESS_KEY_ID', value=cos_username))
notebook_op.container.add_env_variable(V1EnvVar(name='AWS_SECRET_ACCESS_KEY', value=cos_password))
notebook_op.container.set_image_pull_policy('Always')
return op
def demo_pipeline():
stats_op = run_notebook_op('stats', 'generate-community-overview')
contributions_op = run_notebook_op('contributions', 'generate-community-contributions')
run_notebook_op('overview', 'overview').after(stats_op, contributions_op)
# Compile the new pipeline
kfp.compiler.Compiler().compile(demo_pipeline,'pipelines/pipeline.tar.gz')
# Upload the compiled pipeline
client = kfp.Client(host=kfp_url)
pipeline_info = client.upload_pipeline('pipelines/pipeline.tar.gz',pipeline_name='pipeline-demo')
# Create a new experiment
experiment = client.create_experiment(name='demo-experiment')
# Create a new run associated with experiment and our uploaded pipeline
run = client.run_pipeline(experiment.id, 'demo-run', pipeline_id=pipeline_info.id)
Generated Kubeflow Pipelines
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kfp-notebook-0.26.0.tar.gz
(13.4 kB
view details)
Built Distribution
File details
Details for the file kfp-notebook-0.26.0.tar.gz.
File metadata
- Download URL: kfp-notebook-0.26.0.tar.gz
- Upload date:
- Size: 13.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.8.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8a0042cb06729f16690bb3d38d64b548a9e52db1c16b542ee5e6b48fc9fab644 |
|
| MD5 | 42e1165f932c0f958d9afd9e98b7c173 |
|
| BLAKE2b-256 | c142fb5de6b6068e5f930d0e5158257b2627c8529799b5ea408e6a1573472c56 |
File details
Details for the file kfp_notebook-0.26.0-py3-none-any.whl.
File metadata
- Download URL: kfp_notebook-0.26.0-py3-none-any.whl
- Upload date:
- Size: 13.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.8.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7d8afde648ef0e9b1247d4352ff6b5524c2a0eed75a1d80a17e4492cfbf117c4 |
|
| MD5 | 0d47ae1cebc5ebc63ffae52cc4781985 |
|
| BLAKE2b-256 | a89ad0a843864bcaebc1f34a6d4ced7104f1a35c1fdaeedfaa0537c5846aef13 |
