Skip to main content

Jupyter metakernel for apache spark and scala

Project description

# spylon-kernel
[![Build Status](https://travis-ci.org/maxpoint/spylon-kernel.svg?branch=master)](https://travis-ci.org/maxpoint/spylon-kernel)
[![codecov](https://codecov.io/gh/maxpoint/spylon-kernel/branch/master/graph/badge.svg)](https://codecov.io/gh/maxpoint/spylon-kernel)

A Scala [Jupyter kernel](http://jupyter.readthedocs.io/en/latest/projects/kernels.html) that uses [metakernel](https://github.com/Calysto/metakernel) in combination with [py4j](https://www.py4j.org/).

## Prerequisites

* Apache Spark 2.1.1 compiled for Scala 2.11
* Jupyter Notebook
* Python 3.5+

## Install

You can install the spylon-kernel package using `pip` or `conda`.

```bash
pip install spylon-kernel
# or
conda install -c conda-forge spylon-kernel
```

## Using it as a Scala Kernel

You can use spylon-kernel as Scala kernel for Jupyter Notebook. Do this when you want
to work with Spark in Scala with a bit of Python code mixed in.

Create a kernel spec for Jupyter notebook by running the following command:

```bash
python -m spylon_kernel install
```

Launch `jupyter notebook` and you should see a `spylon-kernel` as an option
in the *New* dropdown menu.

See [the basic example notebook](./examples/basic_example.ipynb) for information
about how to intiialize a Spark session and use it both in Scala and Python.

## Using it as an IPython Magic

You can also use spylon-kernel as a magic in an IPython notebook. Do this when
you want to mix a little bit of Scala into your primarily Python notebook.

```python
from spylon_kernel import register_ipython_magics
register_ipython_magics()
```

```scala
%%scala
val x = 8
x
```

## Using it as a Library

Finally, you can use spylon-kernel as a Python library. Do this when you
want to evaluate a string of Scala code in a Python script or shell.

```python
from spylon_kernel import get_scala_interpreter

interp = get_scala_interpreter()

# Evaluate the result of a scala code block.
interp.interpret("""
val x = 8
x
""")

interp.last_result()
```

# Release Process

Push a tag and submit a source dist to PyPI.

```
git commit -m 'REL: 0.2.1' --allow-empty
git tag -a 0.2.1 # and enter the same message as the commit
git push origin master # or send a PR

# if everything builds / tests cleanly, release to pypi
make release
```

Then update https://github.com/conda-forge/spylon-kernel-feedstock.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spylon-kernel-0.3.1.tar.gz (33.4 kB view details)

Uploaded Source

File details

Details for the file spylon-kernel-0.3.1.tar.gz.

File metadata

File hashes

Hashes for spylon-kernel-0.3.1.tar.gz
Algorithm Hash digest
SHA256 a9a81526d3cdce371114e43c4d6eeac3a29aa564cbaed2bbc3d75ef837d6479a
MD5 9f3c8b469910933dd3eed09d00add8cd
BLAKE2b-256 ac9650cb71686e2e58229d3404a5f44d44bb83bd57699c47fc341ddd0f58c18f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page