Skip to main content

Jupyter metakernel for apache spark and scala

Project description

# spylon-kernel [![Build Status](https://travis-ci.org/mariusvniekerk/spylon-kernel.svg?branch=master)](https://travis-ci.org/mariusvniekerk/spylon-kernel) [![codecov](https://codecov.io/gh/mariusvniekerk/spylon-kernel/branch/master/graph/badge.svg)](https://codecov.io/gh/mariusvniekerk/spylon-kernel)

This is an extremely early proof of concept for using the metakernel in combination with py4j to make a simpler kernel for scala.

## Installation

On python 3.5+

`bash pip install . `

## Installing the jupyter kernel

To install the jupyter kernel install it using

` python -m spylon_kernel install `

## Using the kernel

The scala spark metakernl prodived a scala kernel by default. At the first scala cell that is run a spark session will be constructed so that a user can interact with the interpreter.

## Customizing the spark context

The launch arguments can be customized using the %%init_spark magic as follows

`python %%init_spark launcher.jars = ["file://some/jar.jar"] launcher.master = "local[4]" launcher.conf.spark.executor.cores = 8 `

## Other languages

Since this makes use of metakernel you can evaluate normal python code using the %%python magic. In addition once the spark context has been created the spark variable will be added to your python ernvironment.

`python %%python df = spark.read.json("examples/src/main/resources/people.json") `

To get completions for python, make sure that you have installed jedi

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spylon-kernel-0.0.1.tar.gz (11.2 kB view details)

Uploaded Source

Built Distribution

spylon_kernel-0.0.1-py2.py3-none-any.whl (13.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file spylon-kernel-0.0.1.tar.gz.

File metadata

File hashes

Hashes for spylon-kernel-0.0.1.tar.gz
Algorithm Hash digest
SHA256 dba22950a5cd4e2113e4e65409fa8e717f63df1732edcfc75748238780d65173
MD5 1f4be97447fa1cf1b02bfb18e95efd6c
BLAKE2b-256 5e095726e9e9e17a9f01fbfa025e6b3bc4fd3615b194826f9ecd6628100a1cb8

See more details on using hashes here.

File details

Details for the file spylon_kernel-0.0.1-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for spylon_kernel-0.0.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 0998b1723dacdf08b8d095dbb744cbd9f9e4dbcf3e3b99478e2cdcda2c28d947
MD5 6fb5def4f98fbdeee72d02b0e6734913
BLAKE2b-256 307a8b5a9b2568c7303bf2cd8a3939e45adc2955428907e6daf7a1accddca90e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page