Skip to main content

{{ DESCRIPTION }}

Project description

DGP UI

This library and app provide a wrapper around airflow, providing a means to add / remove DAGs (Pipelines) via a web-ui based on a configuration defining the Pipeline 'kinds' and the parameters each kind requires.

Pipeline Dashboard

Pipeline Dashboard

Edit/New Pipeline

Edit/New Pipeline

Pipeline Status

Pipeline Status

Quickstart

  1. Create a folder containing:
  • A configuration.yaml file with the details on your pipeline kinds, e.g.
{
    "kinds": [
        {
            "name": "kind1",
            "display": "Kind 1",
            "fields": [
                {
                    "name": "param1",
                    "display": "Parameter 1"
                },
                {
                    "name": "param2",
                    "display": "Parameter 2"
                }
            ]
        },
        {
            "name": "kind2",
            "display": "Kind 2",
            "fields": [
                {
                    "name": "param3",
                    "display": "Parameter 3"
                },
                {
                    "name": "param4",
                    "display": "Parameter 4"
                }
            ]
        }
    ],
    "schedules": [
        {
            "name": "monthly",
            "display": "Monthly"
        },
        {
            "name": "daily",
            "display": "Daily"
        }
    ]

}

(If schedules are not specified, a default schedules list will be used).

  • The Airflow DAGs Creator - a Python file that reads the pipeline configuration and creates your Airflow DAGs. Sample code:
import datetime
import logging
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils import dates
from etl_server.models import Models

etl_models = Models()

default_args = {
    'owner': 'Airflow',
    'depends_on_past': False,
    'start_date': dates.days_ago(1),
}

for pipeline in etl_models.all_pipelines():
  # pipeline looks like this:
  # {
  #   "id": "<identifier>",
  #   "name": "<English Name of Pipeline>",
  #   "kind": "<kind-name>",
  #   "schedule": "<schedule>",
  #   "params": {
  #      "field1": "value1",
  #      .. other fields, based on kind's fields in configuration
  #   }
  # }
    dag_id = pipeline['id']
    logging.info('Initializing DAG %s', dag_id)
    dag = DAG(dag_id, default_args=default_args, schedule_interval=datetime.timedelta(days=1))
    task = BashOperator(task_id=dag_id,
                        bash_command='echo "%s"; sleep 10 ; echo done' % pipeline['name'],
                        dag=dag)
    globals()[dag_id] = dag
  1. Use a docker-compose setup to run the server, an example docker-compose.yaml file:
version: "3"

services:

  db:
    image: postgres:12
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      POSTGRES_DB: etls
    expose:
      - 5432
    volumes: 
      - /var/lib/postgresql/data

  server:
    build: .
    image: akariv/airflow-config-ui
    environment:
      DATABASE_URL: postgresql://postgres:postgres@db/etls
      AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql://postgres:postgres@db/etls
    expose:
      - 5000
    ports:
      - 5000:5000
    depends_on: 
      - db
    volumes: 
      - /path/to/local/dags/folder/:/app/dags

After running (docker-compose up -d server), open your browser at http://localhost:5000 to see the web UI.

Another option is to create a new Docker image which inherits from akariv/airflow-config-ui and replaces the contents of /app/dags/ with the configuration.json file and your DAG Python files.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

etl-server-0.0.13.tar.gz (24.3 kB view details)

Uploaded Source

Built Distribution

etl_server-0.0.13-py3-none-any.whl (28.7 kB view details)

Uploaded Python 3

File details

Details for the file etl-server-0.0.13.tar.gz.

File metadata

  • Download URL: etl-server-0.0.13.tar.gz
  • Upload date:
  • Size: 24.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.4

File hashes

Hashes for etl-server-0.0.13.tar.gz
Algorithm Hash digest
SHA256 aad63e31592d11fc3e0aa3fe5e0265950f1408dd6198d1b37d0aa334a3e1d40c
MD5 9a0813defcd785bd54ba362afd7d72f9
BLAKE2b-256 3ade55b53ab63d8f2ebe55c7416c362dd01c990cf526c2da287b72dee6830d6c

See more details on using hashes here.

File details

Details for the file etl_server-0.0.13-py3-none-any.whl.

File metadata

  • Download URL: etl_server-0.0.13-py3-none-any.whl
  • Upload date:
  • Size: 28.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.4

File hashes

Hashes for etl_server-0.0.13-py3-none-any.whl
Algorithm Hash digest
SHA256 6da904778e69c6932c77717434ebed7dc4241af9377b471252f05a7fffc294c8
MD5 79463de10c02157806c3c3cf5909ec23
BLAKE2b-256 879dff6a0ef6a757b668038437955224ef3247ebe03da19dd39e069cbd854d52

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page