Skip to main content

Jupyter metakernel for apache spark and scala

Project description

# spylon-kernel
[![Build Status](https://travis-ci.org/mariusvniekerk/spylon-kernel.svg?branch=master)](https://travis-ci.org/mariusvniekerk/spylon-kernel)
[![codecov](https://codecov.io/gh/mariusvniekerk/spylon-kernel/branch/master/graph/badge.svg)](https://codecov.io/gh/mariusvniekerk/spylon-kernel)

This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler
kernel for scala.

## Installation

On python 3.5+

```bash
pip install .
```

## Installing the jupyter kernel

To install the jupyter kernel install it using

```
python -m spylon_kernel install
```

## Using the kernel

The scala spark metakernl prodived a scala kernel by default.
At the first scala cell that is run a spark session will be constructed so that a user can interact with the
interpreter.

### Customizing the spark context

The launch arguments can be customized using the `%%init_spark` magic as follows

```python
%%init_spark
launcher.jars = ["file://some/jar.jar"]
launcher.master = "local[4]"
launcher.conf.spark.executor.cores = 8
```

### Other languages

Since this makes use of metakernel you can evaluate normal python code using the `%%python` magic. In addition once
the spark context has been created the `spark` variable will be added to your python ernvironment.

```python
%%python
df = spark.read.json("examples/src/main/resources/people.json")
```

To get completions for python, make sure that you have installed `jedi`

## Using as a magic

Spylon-kernel can be used as a magic in an existing ipykernel. This is the recommended solution when you want to write
relatively small blocks of scala.

```python
from spylon_kernel import register_ipython_magics
register_ipython_magics()
```

```scala
%%scala
val x = 8
x
```

## Using as a library

If you just want to send a string of scala code to the interpreter and evaluate it you can
do that too.

```python
from spylon_kernel import get_scala_interpreter

interp = get_scala_interpreter()

# Evaluate the result of a scala code block.
interp.interpret("""
val x = 8
x
""")

interp.last_result()
```

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spylon-kernel-0.1.2.tar.gz (30.8 kB view details)

Uploaded Source

File details

Details for the file spylon-kernel-0.1.2.tar.gz.

File metadata

File hashes

Hashes for spylon-kernel-0.1.2.tar.gz
Algorithm Hash digest
SHA256 1e2117573f23d25c22e0323812b7a62bcbc9b57a11ecf4864389854675b6281b
MD5 4e0ce0ae2729b9e1daee0c81116ec93d
BLAKE2b-256 7a194a57677a2ab0badb4f2fb63ef859518a99ea5500fda6922e72dafeb8702f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page