Skip to main content

Wrapper for Great Expectations to fit the requirements of the Gemeente Amsterdam.

Project description

Introduction

This repository contains functions that will ease the use of Great Expectations. Users can input data and data quality rules and get results in return.

DISCLAIMER: The package is in MVP phase

Getting started

Install the dq suite on your compute, for example by running the following code in your workspace:

pip install dq-suite-amsterdam

To validate your first table:

  • define dq_rule_json_path as a path to a JSON file, similar to shown in dq_rules_example.json in this repo
  • define table_name as the name of the table for which a data quality check is required. This name should also occur in the JSON file
  • load the table requiring a data quality check into a PySpark dataframe df (e.g. via spark.read.csv or spark.read.table)
import dq_suite

validation_settings_obj = dq_suite.ValidationSettings(spark_session=spark, 
                                                      catalog_name="dpxx_dev",
                                                      table_name=table_name,
                                                      check_name="name_of_check_goes_here")
dq_suite.run(json_path=dq_rule_json_path, df=df, validation_settings_obj=validation_settings_obj)

Looping over multiple data frames may require a redefinition of the json_path and validation_settings variables.

See the documentation of ValidationSettings for what other parameters can be passed upon intialisation (e.g. Slack or MS Teams webhooks for notifications, location for storing GX, etc).

Create data quality schema and tables (in respective catalog of data team)

for the first time installation create data quality schema and tables from the notebook from repo path scripts/data_quality_tables.sql

  • open the notebook, connect to a cluster
  • select the catalog of the data team and execute the notebook. It will check if schema is available if not it will create schema and same for tables.

Export the schema from Unity Catalog to the Input Form

In order to output the schema from Unity Catalog, use the following commands (using the required schema name):

schema_output = dq_suite.export_schema('schema_name', spark)
print(schema_output)

Copy the string to the Input Form to quickly ingest the schema in Excel.

Validate the schema of a table

It is possible to validate the schema of an entire table to a schema definition from Amsterdam Schema in one go. This is done by adding two fields to the "dq_rules" JSON when describing the table (See: https://github.com/Amsterdam/dq-suite-amsterdam/blob/main/dq_rules_example.json).

You will need:

  • validate_table_schema: the id field of the table from Amsterdam Schema
  • validate_table_schema_url: the url of the table or dataset from Amsterdam Schema

The schema definition is converted into column level expectations (expect_column_values_to_be_of_type) on run time.

Known exceptions

  • The functions can run on Databricks using a Personal Compute Cluster or using a Job Cluster. Using a Shared Compute Cluster will result in an error, as it does not have the permissions that Great Expectations requires.

  • Since this project requires Python >= 3.10, the use of Databricks Runtime (DBR) >= 13.3 is needed (click). Older versions of DBR will result in errors upon install of the dq-suite-amsterdam library.

Contributing to this library

See the separate developers' readme.

Updates

Version 0.1: Run a DQ check for a dataframe

Version 0.2: Run a DQ check for multiple dataframes

Version 0.3: Refactored I/O

Version 0.4: Added schema validation with Amsterdam Schema per table

Version 0.5: Export schema from Unity Catalog

Version 0.6: The results are written to tables in the "dataquality" schema

Version 0.7: Refactored the solution

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dq_suite_amsterdam-0.7.6.tar.gz (14.3 kB view details)

Uploaded Source

Built Distribution

dq_suite_amsterdam-0.7.6-py3-none-any.whl (13.9 kB view details)

Uploaded Python 3

File details

Details for the file dq_suite_amsterdam-0.7.6.tar.gz.

File metadata

  • Download URL: dq_suite_amsterdam-0.7.6.tar.gz
  • Upload date:
  • Size: 14.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.5

File hashes

Hashes for dq_suite_amsterdam-0.7.6.tar.gz
Algorithm Hash digest
SHA256 74431422d50419e6f8d013ca06a009f2c8fed15e4ce898bc0ed30b4a615aa7aa
MD5 6b1e38007f0866be48c14a8c52585f0f
BLAKE2b-256 91fc728d29304b6dfd41cb9ed88979263ee6d5d819ddcf77b401154eed0b6ce6

See more details on using hashes here.

File details

Details for the file dq_suite_amsterdam-0.7.6-py3-none-any.whl.

File metadata

File hashes

Hashes for dq_suite_amsterdam-0.7.6-py3-none-any.whl
Algorithm Hash digest
SHA256 4f26fd6fa79c12d8a73714caa50f52b74311c351fabfa9b0c91c124f238f5064
MD5 55ac086a20c4679e27e49ccf08ecec62
BLAKE2b-256 e28eb8e3774d44bc94bae931e21ad0ab50cc9617a3fa9f5588d4a4d35f14255d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page