An Apache Airflow provider for Great Expectations
Project description
Apache Airflow Provider for Great Expectations
Experimental library as of February 2021
An Airflow operator for Great Expectations, a Python library for testing and validating data.
Installation
Pre-requisites: An environment running great_expectations and apache-airflow, of course.
pip install airflow-provider-great-expectations
In order to run the BigQueryOperator, you will also need to install the relevant dependencies: pybigquery and apache-airflow-providers-google
Modules
Great Expectations Operator: A base operator for Great Expectations. Import into your DAG via:
from great_expectations_provider.operators.great_expectations import GreatExpectationsOperator
Great Expectations BigQuery Operator: An operator for Great Expectations that provides some pre-set parameters for a BigQuery Datasource and Expectation, Validation, and Data Docs stores in Google Cloud Storage. The operator can also be configured to send email on validation failure. See the docstrings in the class for more configuration options. Import into your DAG via:
from great_expectations_provider.operators.great_expectations_bigquery import GreatExpectationsBigQueryOperator
Examples
See the examples directory for an example DAG with some sample tasks that demonstrate operator functionality. The example DAG file contains a comment with instructions on how to run the examples.
**This operator is in very early stages of development! Feel free to submit issues, PRs, or ping the current author (Sam Bail) in the Great Expectations Slack for feedback. Thanks to Pete DeJoy and the Astronomer.io team for the support.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for airflow-provider-great-expectations-0.0.4.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d0a77fca005977cbea0ce0151b76f0892ada90ea13324ea15e96a53e939b6ac6 |
|
| MD5 | eb07029711f4c8560f1e4b994c64bebd |
|
| BLAKE2b-256 | f48ca0c16a6af302d7af93d06dfbc5865dc81a948a1568498d0af63678fee579 |
