The portable Python dataframe library

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for Charles.Cloud from gravatar.com Charles.Cloud Avatar for codydkdc from gravatar.com codydkdc Avatar for gforsyth from gravatar.com gforsyth Avatar for jcrist from gravatar.com jcrist Avatar for jreback from gravatar.com jreback Avatar for wesm from gravatar.com wesm

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

Ibis

Documentation status Project chat Anaconda badge PyPI Build status Build status Codecov branch

What is Ibis?

Ibis is the portable Python dataframe library:

See the documentation on "Why Ibis?" to learn more.

Getting started

You can pip install Ibis with a backend and example data:

pip install 'ibis-framework[duckdb,examples]'

๐Ÿ’ก Tip

See the installation guide for more installation options.

Then use Ibis:

>>> import ibis
>>> ibis.options.interactive = True
>>> t = ibis.examples.penguins.fetch()
>>> t
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ species โ”ƒ island    โ”ƒ bill_length_mm โ”ƒ bill_depth_mm โ”ƒ flipper_length_mm โ”ƒ body_mass_g โ”ƒ sex    โ”ƒ year  โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ string  โ”‚ string    โ”‚ float64        โ”‚ float64       โ”‚ int64             โ”‚ int64       โ”‚ string โ”‚ int64 โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Adelie  โ”‚ Torgersen โ”‚           39.1 โ”‚          18.7 โ”‚               181 โ”‚        3750 โ”‚ male   โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           39.5 โ”‚          17.4 โ”‚               186 โ”‚        3800 โ”‚ female โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           40.3 โ”‚          18.0 โ”‚               195 โ”‚        3250 โ”‚ female โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           NULL โ”‚          NULL โ”‚              NULL โ”‚        NULL โ”‚ NULL   โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           36.7 โ”‚          19.3 โ”‚               193 โ”‚        3450 โ”‚ female โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           39.3 โ”‚          20.6 โ”‚               190 โ”‚        3650 โ”‚ male   โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           38.9 โ”‚          17.8 โ”‚               181 โ”‚        3625 โ”‚ female โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           39.2 โ”‚          19.6 โ”‚               195 โ”‚        4675 โ”‚ male   โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           34.1 โ”‚          18.1 โ”‚               193 โ”‚        3475 โ”‚ NULL   โ”‚  2007 โ”‚
โ”‚ Adelie  โ”‚ Torgersen โ”‚           42.0 โ”‚          20.2 โ”‚               190 โ”‚        4250 โ”‚ NULL   โ”‚  2007 โ”‚
โ”‚ โ€ฆ       โ”‚ โ€ฆ         โ”‚              โ€ฆ โ”‚             โ€ฆ โ”‚                 โ€ฆ โ”‚           โ€ฆ โ”‚ โ€ฆ      โ”‚     โ€ฆ โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
>>> g = t.group_by("species", "island").agg(count=t.count()).order_by("count")
>>> g
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ species   โ”ƒ island    โ”ƒ count โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ string    โ”‚ string    โ”‚ int64 โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Adelie    โ”‚ Biscoe    โ”‚    44 โ”‚
โ”‚ Adelie    โ”‚ Torgersen โ”‚    52 โ”‚
โ”‚ Adelie    โ”‚ Dream     โ”‚    56 โ”‚
โ”‚ Chinstrap โ”‚ Dream     โ”‚    68 โ”‚
โ”‚ Gentoo    โ”‚ Biscoe    โ”‚   124 โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

๐Ÿ’ก Tip

See the getting started tutorial for a full introduction to Ibis.

Python + SQL: better together

For most backends, Ibis works by compiling its dataframe expressions into SQL:

>>> ibis.to_sql(g)
SELECT
  "t1"."species",
  "t1"."island",
  "t1"."count"
FROM (
  SELECT
    "t0"."species",
    "t0"."island",
    COUNT(*) AS "count"
  FROM "penguins" AS "t0"
  GROUP BY
    1,
    2
) AS "t1"
ORDER BY
  "t1"."count" ASC

You can mix SQL and Python code:

>>> a = t.sql("SELECT species, island, count(*) AS count FROM penguins GROUP BY 1, 2")
>>> a
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ species   โ”ƒ island    โ”ƒ count โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ string    โ”‚ string    โ”‚ int64 โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Adelie    โ”‚ Torgersen โ”‚    52 โ”‚
โ”‚ Adelie    โ”‚ Biscoe    โ”‚    44 โ”‚
โ”‚ Adelie    โ”‚ Dream     โ”‚    56 โ”‚
โ”‚ Gentoo    โ”‚ Biscoe    โ”‚   124 โ”‚
โ”‚ Chinstrap โ”‚ Dream     โ”‚    68 โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
>>> b = a.order_by("count")
>>> b
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”“
โ”ƒ species   โ”ƒ island    โ”ƒ count โ”ƒ
โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”ฉ
โ”‚ string    โ”‚ string    โ”‚ int64 โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Adelie    โ”‚ Biscoe    โ”‚    44 โ”‚
โ”‚ Adelie    โ”‚ Torgersen โ”‚    52 โ”‚
โ”‚ Adelie    โ”‚ Dream     โ”‚    56 โ”‚
โ”‚ Chinstrap โ”‚ Dream     โ”‚    68 โ”‚
โ”‚ Gentoo    โ”‚ Biscoe    โ”‚   124 โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

This allows you to combine the flexibility of Python with the scale and performance of modern SQL.

Backends

Ibis supports 20+ backends:

How it works

Most Python dataframes are tightly coupled to their execution engine. And many databases only support SQL, with no Python API. Ibis solves this problem by providing a common API for data manipulation in Python, and compiling that API into the backendโ€™s native language. This means you can learn a single API and use it across any supported backend (execution engine).

Ibis broadly supports two types of backend:

  1. SQL-generating backends
  2. DataFrame-generating backends

Ibis backend types

Portability

To use different backends, you can set the backend Ibis uses:

>>> ibis.set_backend("duckdb")
>>> ibis.set_backend("polars")
>>> ibis.set_backend("datafusion")

Typically, you'll create a connection object:

>>> con = ibis.duckdb.connect()
>>> con = ibis.polars.connect()
>>> con = ibis.datafusion.connect()

And work with tables in that backend:

>>> con.list_tables()
['penguins']
>>> t = con.table("penguins")

You can also read from common file formats like CSV or Apache Parquet:

>>> t = con.read_csv("penguins.csv")
>>> t = con.read_parquet("penguins.parquet")

This allows you to iterate locally and deploy remotely by changing a single line of code.

๐Ÿ’ก Tip

Check out the blog on backend agnostic arrays for one example using the same code across DuckDB and BigQuery.

Community and contributing

Ibis is an open source project and welcomes contributions from anyone in the community.

Join our community by interacting on GitHub or chatting with us on Zulip.

For more information visit https://ibis-project.org/.

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for Charles.Cloud from gravatar.com Charles.Cloud Avatar for codydkdc from gravatar.com codydkdc Avatar for gforsyth from gravatar.com gforsyth Avatar for jcrist from gravatar.com jcrist Avatar for jreback from gravatar.com jreback Avatar for wesm from gravatar.com wesm

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

10.0.0.dev256 pre-release

10.0.0.dev231 pre-release

10.0.0.dev200 pre-release

10.0.0.dev176 pre-release

10.0.0.dev148 pre-release

10.0.0.dev145 pre-release

10.0.0.dev141 pre-release

10.0.0.dev125 pre-release

10.0.0.dev120 pre-release

10.0.0.dev97 pre-release

10.0.0.dev93 pre-release

10.0.0.dev91 pre-release

This version

10.0.0.dev77 pre-release

10.0.0.dev71 pre-release

10.0.0.dev69 pre-release

10.0.0.dev55 pre-release

10.0.0.dev52 pre-release

10.0.0.dev49 pre-release

10.0.0.dev35 pre-release

10.0.0.dev30 pre-release

10.0.0.dev27 pre-release

10.0.0.dev25 pre-release

10.0.0.dev18 pre-release

10.0.0.dev6 pre-release

9.5.0

9.4.0

9.3.0

9.2.0

9.1.0

9.0.0

9.0.0.dev686 pre-release

9.0.0.dev665 pre-release

9.0.0.dev619 pre-release

9.0.0.dev551 pre-release

9.0.0.dev511 pre-release

9.0.0.dev457 pre-release

9.0.0.dev413 pre-release

9.0.0.dev356 pre-release

9.0.0.dev311 pre-release

9.0.0.dev272 pre-release

9.0.0.dev34 pre-release

8.0.0

8.0.0.dev259 pre-release

8.0.0.dev210 pre-release

8.0.0.dev178 pre-release

8.0.0.dev148 pre-release

8.0.0.dev118 pre-release

8.0.0.dev90 pre-release

8.0.0.dev78 pre-release

7.2.0

7.1.0

7.0.0

7.0.0.dev627 pre-release

7.0.0.dev525 pre-release

7.0.0.dev501 pre-release

7.0.0.dev439 pre-release

7.0.0.dev230 pre-release

6.2.0

6.1.1.dev176 pre-release

6.1.1.dev135 pre-release

6.1.1.dev96 pre-release

6.1.1.dev22 pre-release

6.1.0

6.0.1.dev128 pre-release

6.0.1.dev58 pre-release

6.0.1.dev33 pre-release

6.0.0

5.1.1.dev596 pre-release

5.1.1.dev554 pre-release

5.1.1.dev516 pre-release

5.1.1.dev467 pre-release

5.1.0

5.0.0

4.1.0

4.0.0

3.2.0

3.1.0

3.0.2

3.0.1 yanked

Reason this release was yanked:

Superseded by 3.0.2

3.0.0 yanked

Reason this release was yanked:

ibis.__version__ is incorrect

2.1.1

2.1.0

2.0.0

1.4.0

1.3.0

1.2.0

1.1.0

1.0.0

0.14.0

0.13.0

0.12.0

0.11.2

0.11.1

0.10.0

0.9.0

0.8.1

0.8.0

0.7.1

0.7.0

0.6.1

v0.6.0

0.5.2

0.5.1

0.4.1

0.4.0

0.3.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ibis_framework-10.0.0.dev77.tar.gz (1.2 MB view details)

Uploaded Source

Built Distribution

ibis_framework-10.0.0.dev77-py3-none-any.whl (1.9 MB view details)

Uploaded Python 3

File details

Details for the file ibis_framework-10.0.0.dev77.tar.gz.

File metadata

  • Download URL: ibis_framework-10.0.0.dev77.tar.gz
  • Upload date:
  • Size: 1.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.14 Linux/6.5.0-1022-azure

File hashes

Hashes for ibis_framework-10.0.0.dev77.tar.gz
Algorithm Hash digest
SHA256 3aedbdb6286ad1a32a21014405436ea3b92a81eb22cb3a25689521ae6bfbeaea
MD5 dea4f8a97dbf9a933cdad5fa6d9dba69
BLAKE2b-256 79d7f41ec0bfb609e22b58144ca72a4519f01b4905caa8fdedaa459f4cc2fef2

See more details on using hashes here.

File details

Details for the file ibis_framework-10.0.0.dev77-py3-none-any.whl.

File metadata

File hashes

Hashes for ibis_framework-10.0.0.dev77-py3-none-any.whl
Algorithm Hash digest
SHA256 151dc1c412cf1b72f670263e3f37a72b896cbc360ab44b8cd71f43d38dcee658
MD5 ee4abe0b7b4c4e4ca99586a67a722b4e
BLAKE2b-256 5c05968146eb2dabec7bb1376a82ec5bf580ccab8314a261ac2e724ab12aaa90

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page