Skip to main content

Koalas: pandas API on Apache Spark

Project description

NOTE: Koalas supports Apache Spark 3.1 and below as it will be officially included to PySpark in the upcoming Apache Spark 3.2. This repository is now in maintenance mode. For Apache Spark 3.2 and above, please use PySpark directly.

pandas API on Apache Spark
Explore Koalas docs »

Live notebook · Issues · Mailing list
Help Thirsty Koalas Devastated by Recent Fires

The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.

pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:

  • Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
  • Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).

We would love to have you try it and give us feedback, through our mailing lists or GitHub issues.

Try the Koalas 10 minutes tutorial on a live Jupyter notebook here. The initial launch can take up to several minutes.

Github Actions codecov Documentation Status Latest Release Conda Version Binder Downloads

Getting Started

Koalas can be installed in many ways such as Conda and pip.

# Conda
conda install koalas -c conda-forge
# pip
pip install koalas

See Installation for more details.

For Databricks Runtime, Koalas is pre-installed in Databricks Runtime 7.1 and above. Try Databricks Community Edition for free. You can also follow these steps to manually install a library on Databricks.

Lastly, if your PyArrow version is 0.15+ and your PySpark version is lower than 3.0, it is best for you to set ARROW_PRE_0_15_IPC_FORMAT environment variable to 1 manually. Koalas will try its best to set it for you but it is impossible to set it if there is a Spark context already launched.

Now you can turn a pandas DataFrame into a Koalas DataFrame that is API-compliant with the former:

import databricks.koalas as ks
import pandas as pd

pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']})

# Create a Koalas DataFrame from pandas DataFrame
df = ks.from_pandas(pdf)

# Rename the columns
df.columns = ['x', 'y', 'z1']

# Do some operations in place:
df['x2'] = df.x * df.x

For more details, see Getting Started and Dependencies in the official documentation.

Contributing Guide

See Contributing Guide and Design Principles in the official documentation.

FAQ

See FAQ in the official documentation.

Best Practices

See Best Practices in the official documentation.

Koalas Talks and Blogs

See Koalas Talks and Blogs in the official documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

koalas-1.8.2.tar.gz (357.8 kB view details)

Uploaded Source

Built Distribution

koalas-1.8.2-py3-none-any.whl (390.8 kB view details)

Uploaded Python 3

File details

Details for the file koalas-1.8.2.tar.gz.

File metadata

  • Download URL: koalas-1.8.2.tar.gz
  • Upload date:
  • Size: 357.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/0.0.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for koalas-1.8.2.tar.gz
Algorithm Hash digest
SHA256 cd072f1a9ae97e87e85e53a1c8a3097777c76f45504e79782d0acff5282fe586
MD5 423c3b93f51d076df668e1980987d6df
BLAKE2b-256 b6c481dce696d90ac382e31ded2d8ba09205f8cb4633aa0a6fbb23c8d4874047

See more details on using hashes here.

Provenance

File details

Details for the file koalas-1.8.2-py3-none-any.whl.

File metadata

  • Download URL: koalas-1.8.2-py3-none-any.whl
  • Upload date:
  • Size: 390.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/0.0.0 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for koalas-1.8.2-py3-none-any.whl
Algorithm Hash digest
SHA256 ebf00963ac604ee8763ab53ebb028bea3c7732a20cb10f4e52c9ae6a008a749f
MD5 8da40e5c58528a0a2c41eecd04f0c62d
BLAKE2b-256 289ad69cf12ea62116873b427e5843be8ae8431b18f2a0714d6f4eec3ee4cda6

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page