No project description provided
Project description
SQLMesh is a next-generation data transformation and modeling framework that is backwards compatible with dbt. It aims to be easy to use, correct, and efficient.
SQLMesh enables data teams to efficiently run and deploy data transformations written in SQL or Python.
It is more than just a dbt alternative.
Core Features
Get instant SQL impact analysis of your changes, whether in the CLI or in SQLMesh Plan Mode
Virtual Data Environments
- See a full diagram of how Virtual Data Environments work
- Watch this video to learn more
- Plan / Apply workflow like Terraform to understand potential impact of changes
- Automatic column level lineage and data contracts
- Easy to use CI/CD bot
Efficiency and Testing
Running this command will generate a unit test file in the tests/ folder: test_stg_payments.yaml
Runs a live query to generate the expected output of the model
sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5"
# run the unit test
sqlmesh test
MODEL (
name tcloud_demo.stg_payments,
cron '@daily',
grain payment_id,
audits (UNIQUE_VALUES(columns = (
payment_id
)), NOT_NULL(columns = (
payment_id
)))
);
SELECT
id AS payment_id,
order_id,
payment_method,
amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */
'new_column' AS new_column, /* non-breaking change example */
FROM tcloud_demo.seed_raw_payments
test_stg_payments:
model: tcloud_demo.stg_payments
inputs:
tcloud_demo.seed_raw_payments:
- id: 66
order_id: 58
payment_method: coupon
amount: 1800
- id: 27
order_id: 24
payment_method: coupon
amount: 2600
- id: 30
order_id: 25
payment_method: coupon
amount: 1600
- id: 109
order_id: 95
payment_method: coupon
amount: 2400
- id: 3
order_id: 3
payment_method: coupon
amount: 100
outputs:
query:
- payment_id: 66
order_id: 58
payment_method: coupon
amount: 18.0
new_column: new_column
- payment_id: 27
order_id: 24
payment_method: coupon
amount: 26.0
new_column: new_column
- payment_id: 30
order_id: 25
payment_method: coupon
amount: 16.0
new_column: new_column
- payment_id: 109
order_id: 95
payment_method: coupon
amount: 24.0
new_column: new_column
- payment_id: 3
order_id: 3
payment_method: coupon
amount: 1.0
new_column: new_column
- Never builds a table more than once
- Partition-based incremental models
- Unit tests and audits
Take SQL Anywhere
Write SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse.
- Compile time error checking and can transpile 10+ different SQL dialects
- Definitions using simply SQL (no need for redundant and confusing Jinja + YAML)
- Self documenting queries using native SQL Comments
For more information, check out the website and documentation.
Getting Started
Install SQLMesh through pypi by running:
mkdir sqlmesh-example
cd sqlmesh-example
python -m venv .env
source .env/bin/activate
pip install sqlmesh
sqlmesh init duckdb # get started right away with a local duckdb instance
Follow the quickstart guide to learn how to use SQLMesh. You already have a head start!
Join Our Community
We want to ship better data with you. Connect with us in the following ways:
- Join the Tobiko Slack Community to ask questions, or just to say hi!
- File an issue on our GitHub
- Send us an email at hello@tobikodata.com with your questions or feedback
- Read our blog
Contribution
Contributions in the form of issues or pull requests are greatly appreciated. Read more on how to contribute to SQLMesh open source.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file sqlmesh-0.121.1.tar.gz.
File metadata
- Download URL: sqlmesh-0.121.1.tar.gz
- Upload date:
- Size: 24.9 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e2dd95374d2bdfad623afb540cbe126221ac5269f9cf69fd6d27b490f342c249 |
|
| MD5 | ed9ec25e8c684e36ab8d264ee5a04536 |
|
| BLAKE2b-256 | d21cdab85cd3bcd522163f696f6e26f798e16a0ff6e95b625dada37b4480df3e |
Provenance
File details
Details for the file sqlmesh-0.121.1-py3-none-any.whl.
File metadata
- Download URL: sqlmesh-0.121.1-py3-none-any.whl
- Upload date:
- Size: 1.8 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.10.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1f3ed0667a2f3ec4205c9e05725e63fd6e7267e92bc6539161739e108d50fc5d |
|
| MD5 | 05829390e97323f10971720d5040ea0c |
|
| BLAKE2b-256 | e23b4e972485d71d7829dae9e2d73ca55c01a6029b9d34446f59fd4ca4a5b2ac |
