Skip to main content

The Databricks adapter plugin for dbt

Project description

databricks logo dbt logo

Unit Tests Badge Integration Tests Badge

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

The Databricks Lakehouse provides one simple platform to unify all your data, analytics and AI workloads.

dbt-databricks

The dbt-databricks adapter contains all of the code enabling dbt to work with Databricks. This adapter is based off the amazing work done in dbt-spark. Some key features include:

  • Easy setup. No need to install an ODBC driver as the adapter uses pure Python APIs.
  • Open by default. For example, it uses the the open and performant Delta table format by default. This has many benefits, including letting you use MERGE as the the default incremental materialization strategy.
  • Support for Unity Catalog. dbt-databricks>=1.1.1 supports the 3-level namespace of Unity Catalog (catalog / schema / relations) so you can organize and secure your data the way you like.
  • Performance. The adapter generates SQL expressions that are automatically accelerated by the native, vectorized Photon execution engine.

Choosing between dbt-databricks and dbt-spark

If you are developing a dbt project on Databricks, we recommend using dbt-databricks for the reasons noted above.

dbt-spark is an actively developed adapter which works with Databricks as well as Apache Spark anywhere it is hosted e.g. on AWS EMR.

Getting started

Installation

Install using pip:

pip install dbt-databricks

Upgrade to the latest version

pip install --upgrade dbt-databricks

Profile Setup

your_profile_name:
  target: dev
  outputs:
    dev:
      type: databricks
      catalog: [optional catalog name, if you are using Unity Catalog, only available in dbt-databricks>=1.1.1]
      schema: [database/schema name]
      host: [your.databrickshost.com]
      http_path: [/sql/your/http/path]
      token: [dapiXXXXXXXXXXXXXXXXXXXXXXX]

Quick Starts

These following quick starts will get you up and running with the dbt-databricks adapter:

Compatibility

The dbt-databricks adapter has been tested:

  • with Python 3.7 or above.
  • against Databricks SQL and Databricks runtime releases 9.1 LTS and later.

Tips and Tricks

Choosing compute for a Python model

You can override the compute used for a specific Python model by setting the http_path property in model configuration. This can be useful if, for example, you want to run a Python model on an All Purpose cluster, while running SQL models on a SQL Warehouse. Note that this capability is only available for Python models.

def model(dbt, session):
    dbt.config(
      http_path="sql/protocolv1/..."
    )

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt_databricks-1.8.7.tar.gz (64.4 kB view details)

Uploaded Source

Built Distribution

dbt_databricks-1.8.7-py3-none-any.whl (89.2 kB view details)

Uploaded Python 3

File details

Details for the file dbt_databricks-1.8.7.tar.gz.

File metadata

  • Download URL: dbt_databricks-1.8.7.tar.gz
  • Upload date:
  • Size: 64.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for dbt_databricks-1.8.7.tar.gz
Algorithm Hash digest
SHA256 41beae7d548219119aa8298444e41e62af87947275a69deb06aba452a79d0eb9
MD5 65749ea5bf5b89e362bdec673b6d4c48
BLAKE2b-256 ddc362a26603fd9de6e9c4cc983c4832a5cbae56e31c5236231c4affc6213f63

See more details on using hashes here.

File details

Details for the file dbt_databricks-1.8.7-py3-none-any.whl.

File metadata

File hashes

Hashes for dbt_databricks-1.8.7-py3-none-any.whl
Algorithm Hash digest
SHA256 0feea9e7fb44839b9cbf03b039c3d2e1b0bb758028f93748231cc72db5a66bf5
MD5 07d9c0622d067e3e78eb6cf1138a7630
BLAKE2b-256 3899cef5ef659a5191943b7a5fcd462fae2298c5699da25c441f6fdbc86cc25e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page