Skip to main content

Koalas: pandas API on Apache Spark

Project description

pandas API on Apache Spark
Explore Koalas docs »

Live notebook · Issues · Mailing list
Help Thirsty Koalas Devasted by Recent Fires

The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.

pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:

  • Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
  • Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).

This project is currently in beta and is rapidly evolving, with a bi-weekly release cadence. We would love to have you try it and give us feedback, through our mailing lists or GitHub issues.

Try the Koalas 10 minutes tutorial on a live Jupyter notebook here. The initial launch can take up to several minutes.

Github Actions Travis CI codecov Documentation Status Latest Release Conda Version Binder

Getting Started

The recommended way of installing Koalas is Conda as below.

conda install koalas -c conda-forge

You can use not only Conda but also multiple ways to install Koalas. See Installation for full instructions to install Koalas.

If you are a Databricks Runtime user, you can install Koalas using the Libraries tab on the cluster UI, or using dbutils in a notebook as below, for the regular Databricks Runtime.

dbutils.library.installPyPI("koalas")
dbutils.library.restartPython()

Note that Koalas requires Databricks Runtime 5.x or above. In the future, we will package Koalas out-of-the-box in both the regular Databricks Runtime and Databricks Runtime for Machine Learning.

Now you can turn a pandas DataFrame into a Koalas DataFrame that is API-compliant with the former:

import databricks.koalas as ks
import pandas as pd

pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']})

# Create a Koalas DataFrame from pandas DataFrame
df = ks.from_pandas(pdf)

# Rename the columns
df.columns = ['x', 'y', 'z1']

# Do some operations in place:
df['x2'] = df.x * df.x

For more details, see Getting Started and Dependencies in the official documentation.

Contributing Guide

See Contributing Guide and Design Principles in the official documentation.

FAQ

See FAQ in the official documentation.

Best Practices

See Best Practices in the official documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

koalas-0.27.0.tar.gz (245.1 kB view details)

Uploaded Source

Built Distribution

koalas-0.27.0-py3-none-any.whl (868.5 kB view details)

Uploaded Python 3

File details

Details for the file koalas-0.27.0.tar.gz.

File metadata

  • Download URL: koalas-0.27.0.tar.gz
  • Upload date:
  • Size: 245.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2.post20191203 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.5

File hashes

Hashes for koalas-0.27.0.tar.gz
Algorithm Hash digest
SHA256 34c2a813eae76cec03b2431173883a7e453c706e41be6f29fece0a90a8ee2274
MD5 0afef04fca0952f30200d1effe3e1ab9
BLAKE2b-256 a27861e6a509d34a8354a92f8e5104c13ed37498acc3f972de6b080e706a2174

See more details on using hashes here.

Provenance

File details

Details for the file koalas-0.27.0-py3-none-any.whl.

File metadata

  • Download URL: koalas-0.27.0-py3-none-any.whl
  • Upload date:
  • Size: 868.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2.post20191203 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.5

File hashes

Hashes for koalas-0.27.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e55abe86a54f0ece8337b25fb781faccacd79bec8a81fa18af6ffd034a8499c1
MD5 3cb491b892851ee003549dd36001f9ac
BLAKE2b-256 daa1003becf31a86dc976f6b40e4c2f0991453e978fe9312f310b65225613d48

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page