Jupyter Notebook operator for Apache Airflow.
Project description
Airflow-Notebook is a Notebook operator to enable running notebooks as part of an Airflow DAG. This package is installed on the host(s) where Apache Airflow webserver and scheduler applications reside.
Building airflow-notebook
make clean install
Usage
Example below on how to use the airflow operator. This particular DAG was generated with a jinja template in Elyra's pipeline editor.
from airflow import DAG
from airflow_notebook.pipeline import NotebookOp
from airflow.utils.dates import days_ago
# Setup default args with older date to automatically trigger when uploaded
args = {
'project_id': 'untitled-0105163134',
}
dag = DAG(
'untitled-0105163134',
default_args=args,
schedule_interval=None,
start_date=days_ago(1),
description='Created with Elyra 2.0.0.dev0 pipeline editor using untitled.pipeline.',
is_paused_upon_creation=False,
)
notebook_op_6055fdfb_908c_43c1_a536_637205009c79 = NotebookOp(name='notebookA',
namespace='default',
task_id='notebookA',
notebook='notebookA.ipynb',
cos_endpoint='http://endpoint.com:31671',
cos_bucket='test',
cos_directory='untitled-0105163134',
cos_dependencies_archive='notebookA-6055fdfb-908c-43c1-a536-637205009c79.tar.gz',
pipeline_outputs=[
'subdir/A.txt'],
pipeline_inputs=[],
image='tensorflow/tensorflow:2.3.0',
in_cluster=True,
env_vars={'AWS_ACCESS_KEY_ID': 'a_key',
'AWS_SECRET_ACCESS_KEY': 'a_secret_key', 'ELYRA_ENABLE_PIPELINE_INFO': 'True'},
config_file="None",
dag=dag,
)
notebook_op_074355ce_2119_4190_8cde_892a4bc57bab = NotebookOp(name='notebookB',
namespace='default',
task_id='notebookB',
notebook='notebookB.ipynb',
cos_endpoint='http://endpoint.com:31671',
cos_bucket='test',
cos_directory='untitled-0105163134',
cos_dependencies_archive='notebookB-074355ce-2119-4190-8cde-892a4bc57bab.tar.gz',
pipeline_outputs=[
'B.txt'],
pipeline_inputs=[
'subdir/A.txt'],
image='elyra/tensorflow:1.15.2-py3',
in_cluster=True,
env_vars={'AWS_ACCESS_KEY_ID': 'a_key',
'AWS_SECRET_ACCESS_KEY': 'a_secret_key', 'ELYRA_ENABLE_PIPELINE_INFO': 'True'},
config_file="None",
dag=dag,
)
notebook_op_074355ce_2119_4190_8cde_892a4bc57bab << notebook_op_6055fdfb_908c_43c1_a536_637205009c79
notebook_op_68120415_86c9_4dd9_8bd6_b2f33443fcc7 = NotebookOp(name='notebookC',
namespace='default',
task_id='notebookC',
notebook='notebookC.ipynb',
cos_endpoint='http://endpoint.com:31671',
cos_bucket='test',
cos_directory='untitled-0105163134',
cos_dependencies_archive='notebookC-68120415-86c9-4dd9-8bd6-b2f33443fcc7.tar.gz',
pipeline_outputs=[
'C.txt', 'C2.txt'],
pipeline_inputs=[
'subdir/A.txt'],
image='elyra/tensorflow:1.15.2-py3',
in_cluster=True,
env_vars={'AWS_ACCESS_KEY_ID': 'a_key',
'AWS_SECRET_ACCESS_KEY': 'a_secret_key', 'ELYRA_ENABLE_PIPELINE_INFO': 'True'},
config_file="None",
dag=dag,
)
notebook_op_68120415_86c9_4dd9_8bd6_b2f33443fcc7 << notebook_op_6055fdfb_908c_43c1_a536_637205009c79
Generated Airflow DAG
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-notebook-0.0.4.tar.gz.
File metadata
- Download URL: airflow-notebook-0.0.4.tar.gz
- Upload date:
- Size: 9.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.23.0 setuptools/49.6.0.post20201009 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | bdf32b2a4f624d537cc475b09ee804a00b96eb43e15a35de7df7170e52e0b9d3 |
|
| MD5 | dfe1f722508f6e2987e39b375a48ac4c |
|
| BLAKE2b-256 | 3358a6627fe4642d8e98ffeb469a825cf2fce6c58eb3f6e45442b32cd4234841 |
File details
Details for the file airflow_notebook-0.0.4-py3-none-any.whl.
File metadata
- Download URL: airflow_notebook-0.0.4-py3-none-any.whl
- Upload date:
- Size: 10.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.23.0 setuptools/49.6.0.post20201009 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e2d6fa3fa170e63aa9c8897e0484893a7a085410afaf23ab188bc9ea749f222a |
|
| MD5 | c8064f8106839d07035c2bb72269f793 |
|
| BLAKE2b-256 | 1ccc4a0b490439b9b932bb68a8c28afebf0b4edd9c8f327f3fe742a84d14e960 |
