A S3-backed ContentsManager implementation for Jupyter

Project description

S3Contents

A S3 and GCS backed ContentsManager implementation for Jupyter.

It aims to a be a transparent, drop-in replacement for Jupyter standard filesystem-backed storage system. With this implementation of a Jupyter Contents Manager you can save all your notebooks, regular files, directories structure directly to a S3/GCS bucket, this could be on AWS/GCP or a self hosted S3 API compatible like minio.

While there is some implementations of this functionality already available online [2] I wasn't able to make them work in newer Jupyter Notebook installations. This aims to be a better tested one by being highly based on the awesome PGContents[1].

Prerequisites

Write access (valid credentials) to an S3/GCS bucket, this could be on AWS/GCP or a self hosted S3 like minio.

Installation

$ pip install s3contents

Jupyter config

Edit ~/.jupyter/jupyter_notebook_config.py by filling the missing values:

S3

from s3contents import S3ContentsManager

c = get_config()

# Tell Jupyter to use S3ContentsManager for all storage.
c.NotebookApp.contents_manager_class = S3ContentsManager
c.S3ContentsManager.access_key_id = <AWS Access Key ID / IAM Access Key ID>
c.S3ContentsManager.secret_access_key = <AWS Secret Access Key / IAM Secret Access Key>
c.S3ContentsManager.bucket = "<bucket-name>>"

Example for play.minio.io:9000:

from s3contents import S3ContentsManager

c = get_config()

# Tell Jupyter to use S3ContentsManager for all storage.
c.NotebookApp.contents_manager_class = S3ContentsManager
c.S3ContentsManager.access_key_id = "Q3AM3UQ867SPQQA43P2F"
c.S3ContentsManager.secret_access_key = "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
c.S3ContentsManager.endpoint_url = "http://play.minio.io:9000"
c.S3ContentsManager.bucket = "s3contents-demo"
c.S3ContentsManager.prefix = "notebooks/test"

GCP

Note that the file ~/.config/gcloud/application_default_credentials.json assumes a posix system when you did gcloud init

from s3contents import GCSContentsManager

c = get_config(

c.NotebookApp.contents_manager_class = GCSContentsManager
c.GCSContentsManager.project = "<your-project>"
c.GCSContentsManager.token = "~/.config/gcloud/application_default_credentials.json"
c.GCSContentsManager.bucket = "<bucket-name>"

AWS IAM

It is also possible to use IAM Role-based access to the S3 bucket from an Amazon EC2 instance; to do that, just leave access_key_id and secret_access_key set to their default values (None), and ensure that the EC2 instance has an IAM role which provides sufficient permissions for the bucket and the operations necessary.

Project details

Release history Release notifications | RSS feed

0.11.2

Dec 14, 2023

0.11.1

Oct 12, 2023

0.11.0

Jul 25, 2023

0.10.1

May 5, 2023

0.10.0

Apr 2, 2023

0.9.0

Nov 30, 2021

0.7.0

Aug 15, 2021

0.6.3

Apr 16, 2021

0.6.2

Feb 3, 2021

0.6.1

Nov 17, 2020

0.6.0

Sep 16, 2020

0.5.1

May 22, 2020

0.5.0

May 8, 2020

0.4.0

Jan 23, 2020

0.3.3

Jan 3, 2020

0.3.2

Nov 11, 2019

0.3.1

Oct 29, 2019

0.3.0

Sep 3, 2019

0.1.13

May 6, 2019

0.1.12

Nov 14, 2018

0.1.11

Oct 7, 2018

0.1.10

Jun 29, 2018

0.1.9

Jun 29, 2018

This version

0.1.8

Jun 8, 2018

0.1.7

Jun 6, 2018

0.1.5

Jun 5, 2018

0.1.4

Jul 28, 2017

0.1.3

Apr 12, 2017

0.1.2

Mar 4, 2017

0.1.1

Feb 24, 2017

0.1.0

Oct 22, 2016

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3contents-0.1.8.tar.gz (30.8 kB view details)

Uploaded Jun 8, 2018 Source

File details

Details for the file s3contents-0.1.8.tar.gz.

File metadata

Download URL: s3contents-0.1.8.tar.gz
Upload date: Jun 8, 2018
Size: 30.8 kB
Tags: Source
Uploaded using Trusted Publishing? No

File hashes

Hashes for s3contents-0.1.8.tar.gz
Algorithm	Hash digest
SHA256	`cc6ea102d66b3bba2ff193ce4cd330bfe73b258439e5a2c6928b4dd6feb6e528`
MD5	`8ab9eb12ee5ab6a15c79fb83166dbcaa`
BLAKE2b-256	`0145ab0252ebd6cade1cd154978f57814eb065233c36e7f172ab59604caf7138`