Jupyter Notebook operator for Apache Airflow.
Project description
Airflow-Notebook is an Notebook op to enable running notebooks as part of a Airflow DAG. This package is installed on the host(s) where Apache Airflow webserver and scheduler applications reside.
Building airflow-notebook
make clean install
Usage
Example below on how to use the operator for testing purposes.
from airflow import DAG
from datetime import datetime, timedelta
from notebook_op import NotebookOp
def run_notebook_op(op_name, cos_pull_archive, pipeline_outputs, pipeline_inputs):
op = NotebookOp(name=op_name,
namespace='default',
task_id=op_name,
notebook=op_name,
cos_endpoint='http://object.storage:30000',
cos_bucket='test-bucket',
cos_directory='test-directory',
cos_pull_archive=cos_pull_archive,
pipeline_outputs=pipeline_outputs,
pipeline_inputs=pipeline_inputs,
image='elyra/tensorflow:1.15.2-py3',
in_cluster=True,
env_vars={'AWS_ACCESS_KEY_ID': 'minio', 'AWS_SECRET_ACCESS_KEY': 'minio123',
'GITHUB_TOKEN': 'XXXXXXXXXXXXX'},
dag=dag,
)
return op
default_args = {
'start_date': datetime(2020, 1, 1),
'project_id' : '{{ pipeline_name }}',
'retries': 1,
'retry_delay': timedelta(minutes=3),
}
dag = DAG(
'test_dag',
default_args=default_args,
description='A Sample Test DAG',
)
notebook_op_1 = run_notebook_op('generate-community-contributions',
'generate-community-contributions-37215ef9-70b2-450d-b1e0-655df0b96d6e.tar.gz',
'community_contributions.csv',
'None'
)
notebook_op_2 = run_notebook_op('generate-overview',
'generate-overview-5244c826-ca17-493c-aeb8-e7a224f44527.tar.gz',
'community_overview.csv',
'None',
)
notebook_op_3 = run_notebook_op('overview',
'overview-3dbdd4e8-dc24-489e-bb89-f4e6b9b5e52a.tar.gz',
'None',
'community_contributions.csv,community_overview.csv',
)
notebook_op_1 << notebook_op_3
notebook_op_2 << notebook_op_3
Generated Airflow DAG
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-notebook-0.0.3.tar.gz.
File metadata
- Download URL: airflow-notebook-0.0.3.tar.gz
- Upload date:
- Size: 9.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.24.0 setuptools/49.6.0.post20201009 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a21464dd71154210563e1c3fc23befa9d442e68a086bb2e00cc4a99c0c13d79a |
|
| MD5 | 41b51cd428bd571e39c08a38f0ebb3ba |
|
| BLAKE2b-256 | de5415b464b851bb16bc003a7769a4d0bf5b571dbe31544a3dbddc08195c79d6 |
File details
Details for the file airflow_notebook-0.0.3-py3-none-any.whl.
File metadata
- Download URL: airflow_notebook-0.0.3-py3-none-any.whl
- Upload date:
- Size: 10.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.24.0 setuptools/49.6.0.post20201009 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.7.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 566e986594897e4a5a0adb94a1c0d8a79ef976b8a713ff0c8c3a36d57d0fbc29 |
|
| MD5 | b9c6ba8e1c7e5612b40811439234f92d |
|
| BLAKE2b-256 | e8ab2bb23d7e8155a26601ae74e9588df84d453ce647589fe61f9330ba56f4f8 |
