Jupyter metakernel for apache spark and scala
Project description
# spylon-kernel
[![Build Status](https://travis-ci.org/mariusvniekerk/spylon-kernel.svg?branch=master)](https://travis-ci.org/mariusvniekerk/spylon-kernel)
[![codecov](https://codecov.io/gh/mariusvniekerk/spylon-kernel/branch/master/graph/badge.svg)](https://codecov.io/gh/mariusvniekerk/spylon-kernel)
This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler
kernel for scala.
## Installation
On python 3.5+
```bash
pip install .
```
## Installing the jupyter kernel
To install the jupyter kernel install it using
```
python -m spylon_kernel install
```
## Using the kernel
The scala spark metakernl prodived a scala kernel by default.
At the first scala cell that is run a spark session will be constructed so that a user can interact with the
interpreter.
### Customizing the spark context
The launch arguments can be customized using the `%%init_spark` magic as follows
```python
%%init_spark
launcher.jars = ["file://some/jar.jar"]
launcher.master = "local[4]"
launcher.conf.spark.executor.cores = 8
```
### Other languages
Since this makes use of metakernel you can evaluate normal python code using the `%%python` magic. In addition once
the spark context has been created the `spark` variable will be added to your python ernvironment.
```python
%%python
df = spark.read.json("examples/src/main/resources/people.json")
```
To get completions for python, make sure that you have installed `jedi`
## Using as a magic
Spylon-kernel can be used as a magic in an existing ipykernel. This is the recommended solution when you want to write
relatively small blocks of scala.
```python
from spylon_kernel import register_ipython_magics
register_ipython_magics()
```
```scala
%%scala
val x = 8
x
```
## Using as a library
If you just want to send a string of scala code to the interpreter and evaluate it you can
do that too.
```python
from spylon_kernel import get_scala_interpreter
interp = get_scala_interpreter()
# Evaluate the result of a scala code block.
interp.interpret("""
val x = 8
x
""")
interp.last_result()
```
[![Build Status](https://travis-ci.org/mariusvniekerk/spylon-kernel.svg?branch=master)](https://travis-ci.org/mariusvniekerk/spylon-kernel)
[![codecov](https://codecov.io/gh/mariusvniekerk/spylon-kernel/branch/master/graph/badge.svg)](https://codecov.io/gh/mariusvniekerk/spylon-kernel)
This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler
kernel for scala.
## Installation
On python 3.5+
```bash
pip install .
```
## Installing the jupyter kernel
To install the jupyter kernel install it using
```
python -m spylon_kernel install
```
## Using the kernel
The scala spark metakernl prodived a scala kernel by default.
At the first scala cell that is run a spark session will be constructed so that a user can interact with the
interpreter.
### Customizing the spark context
The launch arguments can be customized using the `%%init_spark` magic as follows
```python
%%init_spark
launcher.jars = ["file://some/jar.jar"]
launcher.master = "local[4]"
launcher.conf.spark.executor.cores = 8
```
### Other languages
Since this makes use of metakernel you can evaluate normal python code using the `%%python` magic. In addition once
the spark context has been created the `spark` variable will be added to your python ernvironment.
```python
%%python
df = spark.read.json("examples/src/main/resources/people.json")
```
To get completions for python, make sure that you have installed `jedi`
## Using as a magic
Spylon-kernel can be used as a magic in an existing ipykernel. This is the recommended solution when you want to write
relatively small blocks of scala.
```python
from spylon_kernel import register_ipython_magics
register_ipython_magics()
```
```scala
%%scala
val x = 8
x
```
## Using as a library
If you just want to send a string of scala code to the interpreter and evaluate it you can
do that too.
```python
from spylon_kernel import get_scala_interpreter
interp = get_scala_interpreter()
# Evaluate the result of a scala code block.
interp.interpret("""
val x = 8
x
""")
interp.last_result()
```
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spylon-kernel-0.1.2.tar.gz
(30.8 kB
view details)
File details
Details for the file spylon-kernel-0.1.2.tar.gz
.
File metadata
- Download URL: spylon-kernel-0.1.2.tar.gz
- Upload date:
- Size: 30.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e2117573f23d25c22e0323812b7a62bcbc9b57a11ecf4864389854675b6281b |
|
MD5 | 4e0ce0ae2729b9e1daee0c81116ec93d |
|
BLAKE2b-256 | 7a194a57677a2ab0badb4f2fb63ef859518a99ea5500fda6922e72dafeb8702f |