Client library for running Prefect Cloud flows in Saturn Cloud
Project description
prefect-saturn
prefect-saturn
is a Python package that makes it easy to run Prefect Cloud flows on a Dask cluster with Saturn Cloud. For a detailed tutorial, see "Fault-Tolerant Data Pipelines with Prefect Cloud
".
Installation
prefect-saturn
is available on PyPi.
pip install prefect-saturn
prefect-saturn
can be installed directly from GitHub
pip install git+https://github.com/saturncloud/prefect-saturn.git@main
Getting Started
prefect-saturn
is intended for use inside a Saturn Cloud environment, such as a Jupyter notebook.
import prefect
from prefect import Flow, task
from prefect_saturn import PrefectCloudIntegration
@task
def hello_task():
logger = prefect.context.get("logger")
logger.info("hello prefect-saturn")
flow = Flow("sample-flow", tasks=[hello_task])
project_name = "sample-project"
integration = PrefectCloudIntegration(
prefect_cloud_project_name=project_name
)
flow = integration.register_flow_with_saturn(flow)
flow.register(
project_name=project_name,
labels=["saturn-cloud"]
)
Customize Dask
You can customize the size and behavior of the Dask cluster used to run prefect flows. prefect_saturn.PrefectCloudIntegration.register_flow_with_saturn()
accepts to arguments to accomplish this:
dask_cluster_kwargs
: keyword arguments to pass to the constructordask_saturn.SaturnCluster
.dask_adapt_kwargs
: keyword arguments used to configure "Adaptive Scaling"
For example, the code below tells Saturn that this flow should run on a Dask cluster with 3 xlarge workers, and that prefect should shut down the cluster once the flow run has finished.
flow = integration.register_flow_with_saturn(
flow=flow,
dask_cluster_kwargs={
"n_workers": 3,
"worker_size": "xlarge",
"autoclose": True
}
)
flow.register(
project_name=project_name,
labels=["saturn-cloud"]
)
Contributing
See CONTRIBUTING.md
for documentation on how to test and contribute to prefect-saturn
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file prefect-saturn-0.6.0.tar.gz
.
File metadata
- Download URL: prefect-saturn-0.6.0.tar.gz
- Upload date:
- Size: 9.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 517dbf3ae59a2b4c1aef308ec80b2d469e6c7fe9ebd326983b53c6f31d6ce8f7 |
|
MD5 | 6b8e0db37e0d9856cf693535ed35b194 |
|
BLAKE2b-256 | a5ddc48cb958a9d211fed4f6359cac697bee020106b33a20f73ec5b8fcb812e7 |
File details
Details for the file prefect_saturn-0.6.0-py2-none-any.whl
.
File metadata
- Download URL: prefect_saturn-0.6.0-py2-none-any.whl
- Upload date:
- Size: 10.3 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4818968de5838f96008501bb385a5c2996121e97bdf8081d2ba72fd885b8ff41 |
|
MD5 | f6b094936ee64f1b261fce6799d66239 |
|
BLAKE2b-256 | c989dbd9ec6c7f4b3aa8007492bd378d9d4d2755ab5ee21696c4bc5f8c09eaf5 |