pyramid-notebook 0.3.0

How it works

Use cases

• Visualize and analyze your website data.

• Easier and powerful alternative for normal Python prompt - web browser user interface has more candy over old fashioned terminal.

• Share code sessions and recipes with your team mates.

Prerequisites

• Pyramid web site (can be easily extended to other web frameworks)

• Python 3.5+

• OSX, Linux

• uWSGI (on production server only)

Example code

The following is an example how to construct admin_shell view which launches a Notebook for the currently logged in Pyramid user when the view is visited for the first time. For extra security the permission for the notebook view cannot be assigned through normal groups, but the username must be on the whitelist in the INI settings file. This guarantees the shell is initially accessible only by persons who have shell access to the server itself.

For another approach on these views, please see the demo source code.

views.py:

from pyramid.httpexceptions import HTTPFound
from pyramid.view import view_config
from pyramid_notebook import startup
from pyramid_notebook.views import launch_notebook
from pyramid_notebook.views import shutdown_notebook as _shutdown_notebook
from pyramid_notebook.views import notebook_proxy as _notebook_proxy
from pyramid_web20.models import Base


#: Include our database session in notebook so it is easy to query stuff right away from the prompt
SCRIPT = """
from pyramid_web20.models import DBSession as session
"""


GREETING="""
* **session** - SQLAlchemy database session
"""


@view_config(route_name="notebook_proxy", permission="shell")
def notebook_proxy(request):
    """Proxy IPython Notebook requests to the upstream server."""
    return _notebook_proxy(request, request.user.username)


@view_config(route_name="admin_shell", permission="shell")
def admin_shell(request):
    """Open admin shell with default parameters for the user."""
    # Make sure we have a logged in user
    nb = {}

    # Pass around the Pyramid configuration we used to start this application
    global_config = request.registry.settings["pyramid_web20.global_config"]

    # Get the reference to our Pyramid app config file and generate Notebook
    # bootstrap startup.py script for this application
    config_file = global_config["__file__"]
    startup.make_startup(nb, config_file)
    startup.add_script(nb, SCRIPT)
    startup.add_greeting(nb, GREETING)

    #: Include all our SQLAlchemy models in the notebook variables
    startup.include_sqlalchemy_models(nb, Base)

    return launch_notebook(request, request.user.username, notebook_context=nb)


@view_config(route_name="shutdown_notebook", permission="shell")
def shutdown_notebook(request):
    """Shutdown the notebook of the current user."""
    _shutdown_notebook(request, request.user.username)
    return HTTPFound(request.route_url("home"))

We also need to capture the INI settings file on the server start up, so that we can pass it forward to IPython Notebook process. In __init__.py:

def main(global_config, **settings):
    settings["pyramid_web20.global_config"] = global_config

Then we have a custom principals handler granting the shell permission for users read from the user whitelist in the configuration file:

def find_groups(userid, request):
    """Get applied groups and other for the user"""

    from horus.interfaces import IUserModel
    user_class = request.registry.queryUtility(IUserModel)

    # Read superuser names from the config
    superusers = aslist(request.registry.settings.get("pyramid_web20.superusers"))

    user = models.DBSession.query(user_class).get(userid)
    if user:
        if user.can_login():
            principals = ['group:{}'.format(g.name) for g in user.groups]

        # Allow superuser permission
        if user.username in superusers or user.email in superusers:
            principals.append("superuser:superuser")

        return principals

    # User not found, user disabled
    return None

We refer to superuser:super in Pyramid site root object:

class Root:

    __acl__ = [
        ...
        (Allow, "superuser:superuser", 'shell'),
    ]

And here is the configuration file bit:

pyramid_web20.superusers =
    mikko@example.com

Pyramid settings

python_notebook reads following parameters from your Pyramid INI configuration file:

# Where we store IPython Notebook runtime and persistent files
# (pid, saved notebooks, etc.).
# Each user will get a personal subfolder in this folder
pyramid_notebook.notebook_folder = /tmp/pyramid_notebook

# Automatically shutdown IPython Notebook kernel
# after his many seconds have elapsed since startup
pyramid_notebook.kill_timeout = 3600

# Websocket proxy launch function.
# This is a view function that upgrades the current HTTP request to Websocket (101 upgrade protocol)
# and starts the web server websocket proxy loop. Currently only uWSGI supported
# (see below).
pyramid_notebook.websocket_proxy =

# For uWSGI in production
# pyramid_notebook.websocket_proxy = pyramid_notebook.uwsgi.serve_websocket

# If you need to server websockets from alternative domain (See below).
# Example value: https://ws.example.com
pyramid_notebook.alternative_domain =

Notebook context parameters

Notebooks can be opened with context sensitive parameters. Some are filled in by the framework, some of those you can set yourself.

• You pass in your Notebook context parameters when you call launch_notebook().

• To have custom context variables change startup script.

• To have different info screen change greeting text

Example of what context information you can pass below:

{

    # Extra Python script executed on notebook startup - this is saved as startup.py
    "startup": ""

    # Markdown text displayed at the beginning of the notebook
    "greeting": ""

    # List of paths where to load IPython Notebook Jinja templates
    # http://ipython.org/ipython-doc/3/config/options/notebook.html
    "extra_template_paths": []

    # Notebook daemon process id - filled it in by the daemon itself
    "pid": 1234,

    # Notebook daemon kill timeout in seconds - filled in by the the daemon itself after parsing command line arguments
    "kill_timeout": 5,

    # Bound localhost port for this notebook - filled in by the daemon itself after parsing command line arguments
    "http_port": 9999,

    # Set Notebook HTTP Allow Origin header to tell where websockets are allowed to connect
    "allow_origin": 'localhost:9999',

    # Override websocket URL
    "websocket_url": 'ws://localhost:9998',

    # Path in URL where Notebook is proxyed, must match notebook_proxy() view
    "notebook_path": '/notebook/',

    # Hash of this context. This is generated automatically from supplied context dictionary if not given. If the hash changes the notebook is restarted with new context data.
    "context_hash": 'foo',
}

Dead man switch

Launched Notebook processes have maximum life time after which they terminate themselves. Currently the termation is unconditional seconds since the start up, but in the future versions this is expected to change to a dead man switchs where the process only terminates itself if there has not been recent activity.

Websocket proxying

IPython Notebook needs two different kind of connections to function properly

• HTTP connection for loading the pages, assets

• Websocket for real-time communication with Notebook kernel

When you run Pyramid’s pserve development server on your local machine and enter the Notebook shell, the websocket connection is made directly to IPython Notebook port bound localhost. This is because pserve does not have any kind of support for websockets. This behavior is controlled by pyramid_notebook.websocket_proxy setting.

On the production server, you usually run a web server which spawns processes to execute WSGI requests, the Python standard for hosting web applications. Unfortunately, like WSGI for HTTP, there doesn’t exist a standard for doing websocket requests in a Python application. Thus, one has to add support for websockets for each web server separately. Currently pyramid_notebook supports the following web servers

• uWSGI

It is ok to have another web server at the front of uWSGI, like Nginx, as these web servers can usually do proxy pass for websocket connections. You might need to add following to your Nginx config:

# include a variable for the upgrade header
map $http_upgrade $connection_upgrade {
    default   upgrade;
    ''        close;
}

server {
    location / {
        include uwsgi_params;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

pyramid_notebook.websocket_proxy = pyramid_notebook.uwsgi.serve_websocket

http-websockets = true

Websocket and reverse proxy services

Reverse proxy services, like CloudFlare <https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets->`_, might give only limited or no support for websockets. This may manifest itself in the form of 400 Bad Request responses from the server because the reverse proxy service strips out Connection: Upgrade HTTP Request header. In this case it is recommended that you serve IPython Notebook from a separate domain where the websocket connection gets unhindered access to your server.

You need to

• Configure your naked web server to respond to an alternative domain name (ws.example.com).

• Configure pyramid_notebook to rewrite notebook URLs to come from the alternative domain:

  pyramid_notebook.alternative_domain = https://ws.example.com

• Pyramid AuthTktAuthenticationPolicy, by default, supports wildcard authentication cookies.

• You can limit the naked domain to expose /notebook/ URLs only.

Local deployment

Production deployment

HTTPS only

pyramid_notebook accepts HTTPS connections only. HTTP connections are unencrypted and leaking information over HTTP could lead severe compromises.

VPN restrictions

You can configure your web server to allow access to /notebook/ URLs from whitelisted IP networks only.

Access restricted servers

You do not need to run pyramid_notebook sessions on the main web servers. You can configure a server with limited data and code separately for running pyramid_notebook.

The access restricted server can have

• Read-only account on the database

• Source code and configuration files containing sensitive secrets removed (HTTPS keys, API tokens, etc.)

Linux containerization

Notebook process can be made to start inside Linux container. Thus, it would still run on the same server, but you can limit the access to file system and network by the kernel. Read more about Linux cgroups.

Two-factor authentication

Consider requiring your website admins to use two-factor authentication to protect against admin credential loss due to malware, keylogging and such nasties. Example two-factor library for Python.

Taking down loose notebooks

In the case the notebook daemon processes get stuck, e.g. by user starting a infinite loop and do not terminate properly, you can take them down.

• Any time you launch a notebook with different context (different parameters) for the user, the prior notebook process gets terminated forcefully

• You can manually terminate all notebook processes. Ex:

  pkill -f notebook_daemon.py

Crashing Notebooks

The following are indication of crashed Notebook process. The following page on Notebook when you try try to start Notebook through web:

Apparently IPython Notebook daemon process is not running for user

… or the IPython Notebook dialog Connecting failed and connecting to kernel does not work.

Notebook has most likely died because of Python exception. There exists a file notebook.stderr.log, one per each user, where you should be able to read traceback what happened.

Debugging Notebook daemon

The notebook daemon can be started from a command line and supports normal UNIX daemon start, stop and fg commands. You need to give mandatory pid file, working folder, HTTP port and kill timeout arguments.

Example how to start Notebook daemon manually:

python $SOMEWHERE/pyramid_notebook/server/notebook_daemon.py fg /tmp/pyramid_notebook/$USER/notebook.pid /tmp/pyramid_notebook/$USER 8899 3600

Seeing startup script exceptions

If the startup script does not populate your Notebook with default variables as you hope, you can always do

• print(locals()) to see what local variables are set

• print(gocals()) to see what global variables are set

• Manually execute startup script inside IPython Notebook, e.g. exec(open("/tmp/pyramid_notebook/user-1/.jupyter/profile_default/startup/startup.py ").read()) (check the actual path by exploring /tmp/pyramid_notebook on your local filesystem).

Tests

Install test dependencies:

pip install -e ".[test]"

Running manual instance:

pserve pyramid_notebook/demo/development.ini --reload

Username is username and password password.

Running tests:

py.test tests --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini

Running a single test:

py.test tests/* --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini -s -k test_notebook_template

Run full test coverage:

py.test tests/* --cov pyramid_notebook --cov-report xml --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini -s -k test_notebook_template

Running uWSGI server with websockets:

uwsgi --virtualenv=venv --wsgi-file=pyramid_notebook/demo/wsgi.py --pythonpath=venv/bin/python uwsgi.ini

Running uWSGI under Nginx for manual websocket proxy testing (OSX):

pkill nginx ; nginx -c `pwd`/nginx.conf
uwsgi --virtualenv=venv --wsgi-file=pyramid_notebook/demo/wsgi.py --pythonpath=venv/bin/python uwsgi-under-nginx.ini

py.test tests --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini -s -k test_start_stop
...
INFO:pyramid_notebook.notebookmanager:Running notebook command: python ./pyramid_notebook/server/notebook_daemon.py start /tmp/pyramid_notebook_tests/testuser1/notebook.pid /tmp/pyramid_notebook_tests/testuser1 40007 60

python ./pyramid_notebook/server/notebook_daemon.py start /tmp/pyramid_notebook_tests/testuser1/notebook.pid /tmp/pyramid_notebook_tests/testuser1 40005 60

Related work

• https://github.com/jupyter/jupyterhub

• https://github.com/Carreau/IPython-notebook-proxy

• https://github.com/UnataInc/ipydra/tree/master/ipydrar

0.3.0 (2018-10-09)

• Upgraded to IPython 7.0.1 / Jupyter.

• Add support to Python 3.6, 3.7 and 3.8

• Remove support to Python 3.4.

• Enforce Python style checks.

• Update README.

0.2.1 (2017-01-29)

• Fixed unnecessary “Enter token” prompt with the latest Jupyter Notebook versions

0.2 (2016-12-06)

• Upgraded to IPython 5.1 / Jupyter

• Better error messages in various situations

• Add custom shutdown command by customizing IPython toolbar menu

0.1.11 (2016-04-18)

• Upgrade to Daemonocle 1.0

0.1.10 (2016-01-31)

• Allow easily override IPython Notebook current working directory.

0.1.9 (2016-01-31)

• Make it possible to override the default bootstrap scripts and greeting for make_startup()

0.1.8 (2016-01-31)

• Adding ws4py as a dependency as it is required for uWSGI which is currently the only supported implementation

0.1.7 (2016-01-16)

• Fixed README reST syntax

0.1.6 (2016-01-16)

• Switch to xdaemonocle fork of Daemonocle library

0.1.5 (2016-01-07)

• Keep IPython Notebook in 3.x series now by setup.py pinnings, as IPython 4.0 is not supported yet

0.1.4 (2015-12-19)

• Fixed relative image links to absolute in README

0.1.3 (2015-12-19)

• Fixing MANIFEST.in and release issues due to setuptools_git not present

0.1.2 (2015-12-19)

• Fixed README markup

• Fixed broken drone.io integration which prevented tests to pass on CI

0.1.1 (2015-12-19)

• Fixed broken setup.py classifiers

0.1 (2015-12-19)

• Initial release.