Apache Airflow Providers containing 3rd party integrations supported natively in Airflow
Project description
A framework for generating Apache Airflow DAGs from other workflows.
Installation
Install and update using pip:
pip install astronomer-cosmosThis only installs dependencies for core provider. To install all dependencies, run:
pip install 'astronomer-cosmos[all]'To only install the dependencies for a specific integration, specify the integration name as extra argument, example to install dbt integration dependencies, run:
pip install 'astronomer-cosmos[dbt]'Extras
Extra Name |
Installation Command |
Dependencies |
|---|---|---|
all |
pip install 'astronomer-cosmos[all]' |
All |
dbt |
pip install 'astronomer-cosmos[dbt]' |
dbt core |
Example Usage
Imagine we have a dbt project located at ./dbt/my_project.
from cosmos.providers.dbt import DbtDag, DbtTaskGroup, DbtTask
# render as a DAG
dag = DbtDag(
project_dir="./dbt/my_project",
dag_id="my_dag",
schedule_interval="@daily",
default_args={"owner": "airflow"},
)
# render as a task group
with DAG("my_dag", default_args={"owner": "airflow"}) as dag:
task_group = DbtTaskGroup(
project_dir="./dbt/my_project",
task_group_id="my_task_group",
)
# render as an individual task
with DAG("my_dag", default_args={"owner": "airflow"}) as dag:
task = DbtTask(
project_dir="./dbt/my_project",
task_id="my_task",
)Principles
Astronomer Cosmos provides a framework for generating Apache Airflow DAGs from other workflows. Every provider comes with two main components:
extractors: These are responsible for extracting the workflow from the provider and converting it into Task and Group objects.
operators: These are used when the workflow is converted into a DAG. They are responsible for executing the tasks in the workflow.
Astronomer Cosmos is not opinionated in the sense that it does not enforce any rendering method. Rather, it comes with the tools to render workflows as Airflow DAGs, task groups, or individual tasks.
Changelog
We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.
Contributing Guide
All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.
A detailed overview an how to contribute can be found in the Contributing Guide.
As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.
Goals for the project
Goal 1
Goal 2
Goal 3
Limitations
List any limitations
License
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file astronomer-cosmos-0.0.5.tar.gz.
File metadata
- Download URL: astronomer-cosmos-0.0.5.tar.gz
- Upload date:
- Size: 13.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4499a93f87822e8d45aa97679cc8e6e0b21295d3d5c53b5d8bbb7a7d58947763 |
|
| MD5 | fdeaef1d12f15c24c1dc236f83766ca9 |
|
| BLAKE2b-256 | 467194646f800372b6887765fa3e6b2cb5eb9cf816c3d5c989acf21a1e415cd2 |
