Skip to main content

ETL tool based on SqlAlchemy for building robust ETL pipelies with high emphasis on high data quality

Project description

PyPI version PyPI Build Status codecov Requirements Status Documentation Get on Slack

SqlTask

SqlTask is an extensible ETL library based on SqlAlchemy to help build robust ETL pipelines with high emphasis on data quality.

Main features of SqlTask:

  • Create well documented data models that support iterative development of both schema and data transformation logic.
  • Tightly coupled data quality checking with transformation logic with automatic creation of visualization-friendly and actionable data quality tables.
  • Make use of SQL where practical, especially expensive data filtering and aggregation during data extraction.
  • Row-by-row data transformation using Python where SQL falls short, e.g. calling third party libraries or storing state from previous rows.
  • Encourage use of modern version control tools and processes, especially GIT.
  • Performant data uploading/insertion where supported.
  • Easy integration with modern ETL orchestration tools, especially Apache Airflow.

Word of caution: SqlTask is currently under heavy development, and the API is expected to change frequently.

Supported databases

SqlTask supports all databases with a SqlAlchemy dialect, with dedicated support for the following engines:

  • Google BigQuery
  • MS SQL Server (experimental)
  • Postgres
  • Sqlite
  • Snowflake

Engines not listed above will fall back to using regular inserts.

Installation instructions

To install SqlTask without any dependencies, simply run

pip install sqltask

To automatically pull in dependencies needed by Snowflake, type

pip install sqltask[snowflake]

Please refer to the documentation on Read The Docs for further information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sqltask-0.5.1.tar.gz (18.7 kB view details)

Uploaded Source

File details

Details for the file sqltask-0.5.1.tar.gz.

File metadata

  • Download URL: sqltask-0.5.1.tar.gz
  • Upload date:
  • Size: 18.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for sqltask-0.5.1.tar.gz
Algorithm Hash digest
SHA256 f206bbf849d77ae9a3ae87b2dcede6833779db648ee61c514a305d333676d29d
MD5 e2d18b1a5af8ba0f9efe61842c91628e
BLAKE2b-256 ead9af2e4fea51d05cdfbfa9ef09e3f750834b767f9e7aca573596e7332e225b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page