Productivity-centric Python Big Data Framework
Project description
Ibis
What is Ibis?
Ibis is a Python library that provides a lightweight, universal interface for data wrangling. It helps Python users explore and transform data of any size, stored anywhere.
Ibis has three primary components:
- A dataframe API for Python. Python users can write Ibis code to manipulate tabular data.
- Interfaces to 15+ query engines. Wherever data is stored, people can use Ibis as their API of choice to communicate with any of those query engines.
- Deferred execution. Ibis uses deferred execution, so execution of code is pushed to the query engine. Users can execute at the speed of their backend, not their local computer.
Why Use Ibis?
Ibis aims to be a future-proof solution to interacting with data using Python and can accomplish this goal through its main features:
- Familiar API: Ibis’s API design borrows from popular APIs like pandas and dplyr that most users already know and like to use.
- Consistent syntax: Ibis aims to be a universal Python API for tabular data of any size, big or small.
- Deferred execution: Ibis pushes code execution to the query engine and only moves required data into memory when necessary. Analytics workflows are faster and more efficient
- Interactive mode: Ibis provides an interactive mode in which users can quickly diagnose problems, explore data, and mock up workflows and pipelines locally.
- 10+ supported backends: Ibis supports multiple query engines and DataFrame APIs. Use one interface to transform with your data wherever it lives: from DataFrames in pandas to Parquet files through DuckDB to tables in BigQuery.
- Minimize rewrites: Teams can often keep their Ibis code the same regardless of backend changes, like increasing or decreasing computing power, changing the number or size of their databases, or switching backends entirely.
- Flexibility when you need it: When Ibis doesn't support something, it provides a way to jump directly into SQL.
Common Use Cases
- Speed up prototype to production. Scale code written and tested locally to a distributed system or cloud SQL engine with minimal rewrites.
- Boost performance of existing Python or pandas code. For example a general rule of thumb for pandas is "Have 5 to 10 times as much RAM as the size of your dataset". When a dataset exceeds this rule using in-memory frameworks like pandas can be slow. Instead, using Ibis will significantly speed up your workflows because of its deferred execution. Ibis also empowers you to switch to a faster database engine, without changing much of your code.
- Get rid of long, error-prone,
f-strings. Ibis provides one syntax for multiple query engines and dataframe APIs that lets you avoid learning new flavors of SQL or other framework-specific code. Learn the syntax once and use that syntax anywhere.
Backends
Ibis acts as a universal frontend to the following systems:
- Apache Arrow DataFusion (experimental)
- Apache Druid (experimental)
- Apache Impala
- Apache PySpark
- BigQuery
- ClickHouse
- Dask
- DuckDB
- HeavyAI
- MySQL
- Oracle (experimental)
- Pandas
- Polars (experimental)
- PostgreSQL
- SQL Server
- SQLite
- Snowflake (experimental)
- Trino (experimental)
The list of supported backends is continuously growing. Anyone can get involved in adding new ones! Learn more about contributing to ibis in our contributing documentation at https://github.com/ibis-project/ibis/blob/master/docs/CONTRIBUTING.md
Installation
Install Ibis from PyPI with:
pip install 'ibis-framework[duckdb]'
Or from conda-forge with:
conda install ibis-framework -c conda-forge
(It’s a common mistake to pip install ibis. If you try to use Ibis and get errors early on try uninstalling ibis and installing ibis-framework)
To discover ibis, we suggest starting with the DuckDB backend (which is included by default in the conda-forge package). The DuckDB backend is performant and fully featured.
To use ibis with other backends, include the backend name in brackets for PyPI:
pip install 'ibis-framework[postgres]'
Or use ibis-$BACKEND where $BACKEND is the specific backend you want to use when installing from conda-forge:
conda install ibis-postgres -c conda-forge
Getting Started with Ibis
We provide a number of tutorial and example notebooks in the
ibis-examples. The easiest way
to try these out is through the online interactive notebook environment
provided here:
You can also get started analyzing any dataset, anywhere with just a few lines of Ibis code. Here’s an example of how to use Ibis with a SQLite database.
Download the SQLite database from the ibis-tutorial-data GCS (Google Cloud
Storage) bucket, then connect to it using ibis.
curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'
Connect to the database and show the available tables
>>> import ibis
>>> from ibis import _
>>> ibis.options.interactive = True
>>> con = ibis.sqlite.connect("geography.db")
>>> con.tables
Tables
------
- countries
- gdp
- independence
Choose the countries table and preview its first few rows
>>> countries = con.tables.countries
>>> countries.head()
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ iso_alpha2 ┃ iso_alpha3 ┃ iso_numeric ┃ fips ┃ name ┃ capital ┃ area_km2 ┃ population ┃ continent ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ string │ string │ int32 │ string │ string │ string │ float64 │ int32 │ string │
├────────────┼────────────┼─────────────┼────────┼──────────────────────┼──────────────────┼──────────┼────────────┼───────────┤
│ AD │ AND │ 20 │ AN │ Andorra │ Andorra la Vella │ 468.0 │ 84000 │ EU │
│ AE │ ARE │ 784 │ AE │ United Arab Emirates │ Abu Dhabi │ 82880.0 │ 4975593 │ AS │
│ AF │ AFG │ 4 │ AF │ Afghanistan │ Kabul │ 647500.0 │ 29121286 │ AS │
│ AG │ ATG │ 28 │ AC │ Antigua and Barbuda │ St. Johns │ 443.0 │ 86754 │ NA │
│ AI │ AIA │ 660 │ AV │ Anguilla │ The Valley │ 102.0 │ 13254 │ NA │
└────────────┴────────────┴─────────────┴────────┴──────────────────────┴──────────────────┴──────────┴────────────┴───────────┘
Show the 5 least populous countries in Asia
>>> (
... countries.filter(_.continent == "AS")
... .select("name", "population")
... .order_by(_.population)
... .limit(5)
... )
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name ┃ population ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ string │ int32 │
├────────────────────────────────┼────────────┤
│ Cocos [Keeling] Islands │ 628 │
│ British Indian Ocean Territory │ 4000 │
│ Brunei │ 395027 │
│ Maldives │ 395650 │
│ Macao │ 449198 │
└────────────────────────────────┴────────────┘
Community and Contributing
Ibis is an open source project and welcomes contributions from anyone in the community.
- Read the contributing guide.
- We care about keeping the community welcoming for all. Check out the code of conduct.
- The Ibis project is open sourced under the Apache License.
Join our community by interacting on GitHub or chatting with us on Zulip.
For more information visit https://ibis-project.org/.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file ibis_framework-7.0.0.dev627.tar.gz.
File metadata
- Download URL: ibis_framework-7.0.0.dev627.tar.gz
- Upload date:
- Size: 1.1 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.10.13 Linux/6.2.0-1012-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 387c19668078f679183e8f2e8a04ccc3f9407ce743b68ca1bada200e45301517 |
|
| MD5 | e8cec1a151987b7cdacf03324a118e59 |
|
| BLAKE2b-256 | 43437dae355b35dce0b9d20384644eec29d8f4c41e13503f3719a3b89b4f61e8 |
File details
Details for the file ibis_framework-7.0.0.dev627-py3-none-any.whl.
File metadata
- Download URL: ibis_framework-7.0.0.dev627-py3-none-any.whl
- Upload date:
- Size: 1.7 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.10.13 Linux/6.2.0-1012-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7ddbb7e6f9654750b2734faf7c63e33bf2e7f764ac2efb138fd4a280267bd037 |
|
| MD5 | 60a828a7237666dc26b6288425cf49e8 |
|
| BLAKE2b-256 | 176adbe8757ee7d6aee4c3dab562849c3170dae49a83d62b56d074535f577d26 |
