Skip to main content

Client library for running Prefect Cloud flows in Saturn Cloud

Project description

prefect-saturn

GitHub Actions PyPI Version

prefect-saturn is a Python package that makes it easy to run Prefect Cloud flows on a Dask cluster with Saturn Cloud. For a detailed tutorial, see "Fault-Tolerant Data Pipelines with Prefect Cloud ".

Installation

prefect-saturn is available on PyPi.

pip install prefect-saturn

prefect-saturn can be installed directly from GitHub

pip install git+https://github.com/saturncloud/prefect-saturn.git@main

Getting Started

prefect-saturn is intended for use inside a Saturn Cloud environment, such as a Jupyter notebook.

import prefect
from prefect import Flow, task
from prefect_saturn import PrefectCloudIntegration


@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("hello prefect-saturn")


flow = Flow("sample-flow", tasks=[hello_task])

project_name = "sample-project"
integration = PrefectCloudIntegration(
    prefect_cloud_project_name=project_name
)
flow = integration.register_flow_with_saturn(flow)

flow.register(
    project_name=project_name,
    labels=["saturn-cloud"]
)

Customize Dask

You can customize the size and behavior of the Dask cluster used to run prefect flows. prefect_saturn.PrefectCloudIntegration.register_flow_with_saturn() accepts to arguments to accomplish this:

For example, the code below tells Saturn that this flow should run on a Dask cluster with 3 xlarge workers, and that prefect should shut down the cluster once the flow run has finished.

flow = integration.register_flow_with_saturn(
    flow=flow,
    dask_cluster_kwargs={
        "n_workers": 3,
        "worker_size": "xlarge",
        "autoclose": True
    }
)

flow.register(
    project_name=project_name,
    labels=["saturn-cloud"]
)

Contributing

See CONTRIBUTING.md for documentation on how to test and contribute to prefect-saturn.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

prefect-saturn-0.6.0.tar.gz (9.1 kB view details)

Uploaded Source

Built Distribution

prefect_saturn-0.6.0-py2-none-any.whl (10.3 kB view details)

Uploaded Python 2

File details

Details for the file prefect-saturn-0.6.0.tar.gz.

File metadata

  • Download URL: prefect-saturn-0.6.0.tar.gz
  • Upload date:
  • Size: 9.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for prefect-saturn-0.6.0.tar.gz
Algorithm Hash digest
SHA256 517dbf3ae59a2b4c1aef308ec80b2d469e6c7fe9ebd326983b53c6f31d6ce8f7
MD5 6b8e0db37e0d9856cf693535ed35b194
BLAKE2b-256 a5ddc48cb958a9d211fed4f6359cac697bee020106b33a20f73ec5b8fcb812e7

See more details on using hashes here.

File details

Details for the file prefect_saturn-0.6.0-py2-none-any.whl.

File metadata

  • Download URL: prefect_saturn-0.6.0-py2-none-any.whl
  • Upload date:
  • Size: 10.3 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.5.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for prefect_saturn-0.6.0-py2-none-any.whl
Algorithm Hash digest
SHA256 4818968de5838f96008501bb385a5c2996121e97bdf8081d2ba72fd885b8ff41
MD5 f6b094936ee64f1b261fce6799d66239
BLAKE2b-256 c989dbd9ec6c7f4b3aa8007492bd378d9d4d2755ab5ee21696c4bc5f8c09eaf5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page