jupyter-project

An JupyterLab extension to handle (a unique) project and files templates. It adds the ability to generate projects from a cookiecutter template as well as generate files from Jinja2 templates. Those templates can be parametrized directly from the frontend by specifying JSON schemas.

This extension is composed of a Python package named jupyter_project for the server extension and a NPM package named jupyter-project for the frontend extension.

Requirements

• Python requirements:

# setup.py#L63-L66

"cookiecutter",
"jinja2~=2.9",
"jsonschema",
"jupyterlab~=1.2"

• Optional Python requirements:

# setup.py#L69-L69

"all": ["jupyter_conda~=3.3"],

• Optional JupyterLab extensions:

  • jupyterlab_conda

Install

Note: You will need NodeJS to install the extension.

pip install jupyter_project
jupyter lab build

Configuring the extension

By default, this extension will not add anything to JupyterLab as the templates must be configured as part of the server extension configuration key JupyterProject (see Jupyter server configuration for more information).

The configuration example for Binder will be described next - this is the file binder/jupyter_notebook_config.json.

The section for this extension must be named JupyterProject:

// ./binder/jupyter_notebook_config.json#L7-L7

"JupyterProject": {

It accepts to optional keys: file_templates and project_template. The first defines a list of places containing templated files. And the second describe the project template. They can both exist alone (i.e. only file templates or only the project template).

The file templates can be located in a location provided by its fullpath or in a location within a Python module. In the Binder example, the template are located in the folder examples part of the jupyter_project Python module:

// ./binder/jupyter_notebook_config.json#L8-L12

"file_templates": [
  {
    "name": "data-sciences",
    "module": "jupyter_project",
    "location": "examples",

The last parameter appearing here is name. It described uniquely the source of file templates.

Than comes the list of templated files available in that source. There are three templated file examples. The shortest configuration is:

// ./binder/jupyter_notebook_config.json#L14-L16

{
  "template": "demo.ipynb"
},

This will create a template by copy of the provided file.

But usually, a template comes with parameters. This extension handles parameters through a JSON schema specification. That schema will be used to prompt the user with a form that will be validated against the schema. Then the form values will be passed to Jinja2 to rendered the templates.

// ./binder/jupyter_notebook_config.json#L74-L92

{
  "default_name": "{{ modelName }}",
  "destination": "src/models",
  "schema": {
    "type": "object",
    "properties": {
      "authorName": {
        "type": "string"
      },
      "modelName": {
        "type": "string",
        "pattern": "^[a-zA-Z_]\\w*$"
      }
    },
    "required": ["modelName"]
  },
  "template_name": "Train Model",
  "template": "train_model.py"
}

In the settings, you can see three additional entries that have not been explained yet:

• template_name: A nicer name for the template to be displayed in the frontend.
• default_name: Default name for the file generated from the template (the string may contain Jinja2 variables defined in the schema).
• destination: If you are using the project template, the generated file will be placed within the destination folder inside the active project folder. If no project is active the file will be written in the current folder.

The latest file template example is a complete example of all possibilities (including type of variables that you could used in the schema):

// ./binder/jupyter_notebook_config.json#L17-L73

{
  "destination": "notebooks",
  "icon": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 16 16\"> <rect class=\"jp-icon3\" fill=\"#ffffff\" width=\"16\" height=\"16\" rx=\"2\" style=\"fill-opacity:1\" /> <path class=\"jp-icon-accent0\" fill=\"#faff00\" d=\"m 12.098275,4.7065364 -4.9999997,-0.62651 v 8.9554396 l 4.9999997,-0.32893 v -1.1 l -3.4999997,0.19305 V 8.9065364 h 1.9999997 v -1.1 l -1.9999997,-0.1 V 5.3539365 l 3.4999997,0.3526 z\" style=\"fill-opacity:1;stroke:none\" /> </svg> ",
  "template_name": "Example",
  "template": "example.ipynb",
  "schema": {
    "type": "object",
    "properties": {
      "exampleBoolean": {
        "default": false,
        "title": "A choice",
        "type": "boolean"
      },
      "exampleList": {
        "default": [1, 2, 3],
        "title": "A list of number",
        "type": "array",
        "items": {
          "default": 0,
          "type": "number"
        }
      },
      "exampleNumber": {
        "default": 42,
        "title": "A number",
        "type": "number",
        "minimum": 0,
        "maximum": 100
      },
      "exampleObject": {
        "default": {
          "number": 1,
          "street_name": "Dog",
          "street_type": "Street"
        },
        "title": "A object",
        "type": "object",
        "properties": {
          "number": { "type": "integer" },
          "street_name": { "type": "string" },
          "street_type": {
            "type": "string",
            "enum": ["Street", "Avenue", "Boulevard"]
          }
        },
        "required": ["number"]
      },
      "exampleString": {
        "default": "I_m_Beautiful",
        "title": "A string",
        "type": "string",
        "pattern": "^[a-zA-Z_]\\w*$"
      }
    },
    "required": ["exampleString"]
  }
},

A careful reader may notice the last available setting: icon. It is a stringified svg that will be used to set a customized icon in the frontend for the template.

If you need to set templates from different sources, you can add entry similar to data-sciences in the file_templates list.

The second major configuration section is project_template. Each template must specified a value for template that points to a valid cookiecutter template source:

// ./binder/jupyter_notebook_config.json#L96-L97

"project_template": {
  "template": "https://github.com/drivendata/cookiecutter-data-science",

The cookiecutter template parameters that you wish the user to be able to change must be specified as a JSON schema:

// ./binder/jupyter_notebook_config.json#L98-L125

"schema": {
  "type": "object",
  "properties": {
    "project_name": {
      "type": "string",
      "default": "Project Name"
    },
    "repo_name": {
      "title": "Folder name",
      "type": "string",
      "pattern": "^[a-zA-Z_]\\w*$",
      "default": "project_name"
    },
    "author_name": {
      "type": "string",
      "description": "Your name (or your organization/company/team)"
    },
    "description": {
      "type": "string",
      "description": "A short description of the project."
    },
    "open_source_license": {
      "type": "string",
      "enum": ["MIT", "BSD-3-Clause", "No license file"]
    }
  },
  "required": ["project_name", "repo_name"]
},

Then you need to set folder_name as the name of the folder resulting from the cookiecutter template. This is a string accepting Jinja2 variables defined in the schema.

The latest option in the example is default_path. This is optional and, if set, it should provide the default path (folder or file) to be opened by JupyterLab once the project has been generated:

// ./binder/jupyter_notebook_config.json#L126-L127

"folder_name": "{{ repo_name }}",
"default_path": "README.md",

Conda environment integration

If the jupyter_conda optional extension is installed and if conda_pkgs is specified in the project_template configuration, then a Conda environment will follow the life cycle of the project; i.e. creation of an environment at project creation, update of the environment when opening a project and deletion at project deletion.

The conda_pkgs setting should be set to a string matching the default environment type of conda environment to be created at project creation (see jupyter_conda labextension for more information). You can also set a packages list separated by space.

The binder example defines:

// ./binder/jupyter_notebook_config.json#L128-L128

"conda_pkgs": "awscli click coverage flake8 ipykernel python-dotenv>=0.5.1 sphinx"

The default conda packages settings is the fallback if environment.yml is absent of the project cookiecutter template.

Full configuration

Here is the description of all server extension settings:

{
  "JupyterProject": {
    "file_templates": {
      "description": "List of file template loaders",
      "type": "array",
      "items": {
        "description": ,
        "type": "object",
        "properties": {
          "location": {
            "description": "Templates path",
            "type": "string"
          },
          "module": {
            "description": "Python package containing the templates 'location' [optional]",
            "type": "string"
          },
          "name": {
            "description": "Templates group name",
            "type": "string"
          },
          "files": {
            "description": "List of template files",
            "type": "array",
            "minItems": 1,
            "items": {
              "type": "object",
              "properties": {
                "default_name": {
                  "description": "Default file name (without extension; support Jinja2 templating using the schema parameters)",
                  "default": "Untitled",
                  "type": "string"
                },
                "destination": {
                  "description": "Relative destination folder [optional]",
                  "type": "string"
                },
                "icon": {
                  "description": "Template icon to display in the frontend [optional]",
                  "default": null,
                  "type": "string"
                },
                "schema": {
                  "description": "JSON schema list describing the templates parameters [optional]",
                  "type": "object"
                },
                "template": {
                  "description": "Template path",
                  "type": "string"
                },
                "template_name" : {
                  "description": "Template name in the UI [optional]",
                  "type": "string"
                }
              },
              "required": ["template"]
            }
          }
        },
        "required": ["files", "location", "name"]
      }
    },
    "project_template": {
      "description": "The project template options",
      "type": "object",
      "properties": {
        "configuration_filename": {
          "description": "Name of the project configuration JSON file [optional]",
          "default": "jupyter-project.json",
          "type": "string"
        },
        "configuration_schema": {
          "description": "JSON schema describing the project configuration file [optional]",
          "default": {
            "type": "object",
            "properties": {"name": {"type": "string"}},
            "required": ["name"],
          },
          "type": "object"
        },
        "conda_pkgs": {
          "default": null,
          "description": "Type of conda environment or space separated list of conda packages (requires `jupyter_conda`) [optional]",
          "type": "string"
        },
        "default_path": {
          "description": "Default file or folder to open; relative to the project root [optional]",
          "type": "string"
        },
        "folder_name": {
          "description": "Project name (support Jinja2 templating using the schema parameters) [optional]",
          "default": "{{ name|lower|replace(' ', '_') }}",
          "type": "string"
        },
        "module": {
          "description": "Python package containing the template [optional]",
          "type": "string"
        },
        "schema": {
          "description": "JSON schema describing the template parameters [optional]",
          "default": {
            "type": "object",
            "properties": {"name": {"type": "string", "pattern": "^[a-zA-Z_]\\w*$"}},
            "required": ["name"],
          },
          "type": "object"
        },
        "template": {
          "description": "Cookiecutter template source",
          "default": null,
          "type": "string"
        }
      },
      "required": ["template"]
    }
  }
}

Troubleshoot

If you are seeing the frontend extension but it is not working, check that the server extension is enabled:

jupyter serverextension list

If the server extension is installed and enabled but you are not seeing the frontend, check the frontend is installed:

jupyter labextension list

If it is installed, try:

jupyter lab clean
jupyter lab build

Contributing

The frontend extension is based on uniforms with its material-ui flavor to handle and display automatic forms from JSON schema.

Install

The jlpm command is JupyterLab's pinned version of yarn that is installed with JupyterLab. You may use yarn or npm in lieu of jlpm below.

# Clone the repo to your local environment
# Move to jupyter-project directory

# Install server extension
pip install -e .[test]
# Register server extension
jupyter serverextension enable --py jupyter_project

# Install dependencies
jlpm
# Build Typescript source
jlpm build
# Link your development version of the extension with JupyterLab
jupyter labextension link .
# Rebuild Typescript source after making changes
jlpm build
# Rebuild JupyterLab after making any changes
jupyter lab build

You can watch the source directory and run JupyterLab in watch mode to watch for changes in the extension's source and automatically rebuild the extension and application.

# Watch the source directory in another terminal tab
jlpm watch
# Run jupyterlab in watch mode in one terminal tab
jupyter lab --watch

To run with an working example, execute jupyter lab from the binder folder to use the local jupyter_notebook_config.json as configuration.

Uninstall

pip uninstall jupyter-project

jupyter labextension uninstall jupyter-project

Changelog

v0.1.0

• Handle Jinja2 file templates
• Handle project cookiecutter template