Jupyter metakernel for apache spark and scala
Project description
This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler kernel for scala.
Installation
On python 3.5+
pip install .Installing the jupyter kernel
To install the jupyter kernel install it using
python -m spylon_kernel install
Using the kernel
The scala spark metakernl prodived a scala kernel by default. At the first scala cell that is run a spark session will be constructed so that a user can interact with the interpreter.
Customizing the spark context
The launch arguments can be customized using the %%init_spark magic as follows
%%init_spark
launcher.jars = ["file://some/jar.jar"]
launcher.master = "local[4]"
launcher.conf.spark.executor.cores = 8Other languages
Since this makes use of metakernel you can evaluate normal python code using the %%python magic. In addition once the spark context has been created the spark variable will be added to your python ernvironment.
%%python
df = spark.read.json("examples/src/main/resources/people.json")To get completions for python, make sure that you have installed jedi
Using as a magic
Spylon-kernel can be used as a magic in an existing ipykernel. This is the recommended solution when you want to write relatively small blocks of scala.
from spylon_kernel import register_ipython_magics
register_ipython_magics()%%scala
val x = 8
xUsing as a library
If you just want to send a string of scala code to the interpreter and evaluate it you can do that too.
from spylon_kernel import get_scala_interpreter
interp = get_scala_interpreter()
# Evaluate the result of a scala code block.
interp.interpret("""
val x = 8
x
""")
interp.last_result()Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file spylon-kernel-0.1.1.tar.gz.
File metadata
- Download URL: spylon-kernel-0.1.1.tar.gz
- Upload date:
- Size: 30.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4647473e02341f2d236eed541733dfbd07cb492526a1755e3443c677bae6d0c9 |
|
| MD5 | a691a125327e7465e790f58914dffd1c |
|
| BLAKE2b-256 | 4aaef68fde5cb20538da3e169b3fe49d7c8db711df899bb5ed8458165d7a535c |
