Python utilities for Manubot: Manuscripts, open and automated

Manubot is a workflow and set of tools for the next generation of scholarly publishing. This repository contains a Python package with several Manubot-related utilities, as described in the usage section below. Package documentation is available at https://manubot.github.io/manubot (auto-generated from the Python source code).

The manubot cite command-line interface retrieves and formats bibliographic metadata for user-supplied persistent identifiers like DOIs or PubMed IDs. The manubot process command-line interface prepares scholarly manuscripts for Pandoc consumption. The manubot process command is used by Manubot manuscripts, which are based off the Rootstock template, to automate several aspects of manuscript generation. See Rootstock's manuscript usage guide for more information.

Note: If you want to experience Manubot by editing an existing manuscript, see https://github.com/manubot/try-manubot. If you want to create a new manuscript, see https://github.com/manubot/rootstock.

To cite the Manubot project or for more information on its design and history, see:

Open collaborative writing with Manubot
Daniel S. Himmelstein, Vincent Rubinetti, David R. Slochower, Dongbo Hu, Venkat S. Malladi, Casey S. Greene, Anthony Gitter
PLOS Computational Biology (2019-06-24) https://doi.org/c7np
DOI: 10.1371/journal.pcbi.1007128 · PMID: 31233491 · PMCID: PMC6611653

The Manubot version of this manuscript is available at https://greenelab.github.io/meta-review/.

Installation

If you are using the manubot Python package as part of a manuscript repository, installation of this package is handled though the Rootstock's environment specification. For other use cases, this package can be installed via pip.

Install the latest release version from PyPI:

pip install --upgrade manubot

Or install from the source code on GitHub, using the version specified by a commit hash:

COMMIT=d2160151e52750895571079a6e257beb6e0b1278
pip install --upgrade git+https://github.com/manubot/manubot@$COMMIT

The --upgrade argument ensures pip updates an existing manubot installation if present.

Usage

Installing the python package creates the manubot command line program. Here is the usage information as per manubot --help:

usage: manubot [-h] [--version] {process,cite,webpage} ...

Manubot: the manuscript bot for scholarly writing

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

subcommands:
  All operations are done through subcommands:

  {process,cite,webpage}
    process             process manuscript content
    cite                citation to CSL command line utility
    webpage             deploy Manubot outputs to a webpage directory tree

Note that all operations are done through the following sub-commands.

Process

The manubot process program is the primary interface to using Manubot. There are two required arguments: --content-directory and --output-directory, which specify the respective paths to the content and output directories. The content directory stores the manuscript source files. Files generated by Manubot are saved to the output directory.

One common setup is to create a directory for a manuscript that contains both the content and output directory. Under this setup, you can run the Manubot using:

manubot process \
  --content-directory=content \
  --output-directory=output

See manubot process --help for documentation of all command line arguments:

usage: manubot process [-h] --content-directory CONTENT_DIRECTORY
                       --output-directory OUTPUT_DIRECTORY
                       [--template-variables-path TEMPLATE_VARIABLES_PATH]
                       [--skip-citations] [--cache-directory CACHE_DIRECTORY]
                       [--clear-requests-cache]
                       [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]

Process manuscript content to create outputs for Pandoc consumption. Performs
bibliographic processing and templating.

optional arguments:
  -h, --help            show this help message and exit
  --content-directory CONTENT_DIRECTORY
                        Directory where manuscript content files are located.
  --output-directory OUTPUT_DIRECTORY
                        Directory to output files generated by this script.
  --template-variables-path TEMPLATE_VARIABLES_PATH
                        Path or URL of a file containing template variables
                        for jinja2. Serialization format is inferred from the
                        file extension, with support for JSON, YAML, and TOML.
                        If the format cannot be detected, the parser assumes
                        JSON. Specify this argument multiple times to read
                        multiple files. Variables can be applied to a
                        namespace (i.e. stored under a dictionary key) like
                        `--template-variables-path=namespace=path_or_url`.
                        Namespaces must match the regex `[a-zA-
                        Z_][a-zA-Z0-9_]*`.
  --skip-citations      Skip citation and reference processing. Specify when
                        using the pandoc-manubot-cite filter. If citation-
                        tags.tsv is found in content, these tags will be
                        inserted in the markdown output using the reference-
                        link syntax for citekey aliases. Appends
                        content/manual-references*.* paths to Pandoc's
                        metadata.bibliography field.
  --cache-directory CACHE_DIRECTORY
                        Custom cache directory. If not specified, caches to
                        output-directory.
  --clear-requests-cache
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr logging

Manual references

Manubot has the ability to rely on user-provided reference metadata rather than generating it. manubot process searches the content directory for files containing manually-provided reference metadata that match the glob manual-references*.*. If a manual reference filename ends with .json or .yaml, it's assumed to contain CSL Data (i.e. Citation Style Language JSON). Otherwise, the format is inferred from the extension and converted to CSL JSON using the pandoc-citeproc --bib2json utility. The standard citation key for manual references is inferred from the CSL JSON id or note field. When no prefix is provided, such as doi:, url:, or raw:, a raw: prefix is automatically added. If multiple manual reference files load metadata for the same standard citation id, precedence is assigned according to descending filename order.

Cite

manubot cite is a command line utility to create CSL JSON items for one or more citation keys. Citation keys should be in the format source:identifier. For example, the following example generates CSL JSON for four references:

manubot cite doi:10.1098/rsif.2017.0387 pmid:29424689 pmcid:PMC5640425 arxiv:1806.05726

The following terminal recording demonstrates the main features of manubot cite:

Additional usage information is available from manubot cite --help:

usage: manubot cite [-h] [--render] [--csl CSL]
                    [--format {plain,markdown,docx,html,jats}]
                    [--output OUTPUT] [--allow-invalid-csl-data]
                    [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                    citekeys [citekeys ...]

Retrieve bibliographic metadata for one or more citation keys.

positional arguments:
  citekeys              One or more (space separated) citation keys to produce
                        CSL for.

optional arguments:
  -h, --help            show this help message and exit
  --render              Whether to render CSL Data into a formatted reference
                        list using Pandoc. Pandoc version 2.0 or higher is
                        required for complete support of available output
                        formats.
  --csl CSL             When --render, specify an XML CSL definition to style
                        references (i.e. Pandoc's --csl option). Defaults to
                        Manubot's style.
  --format {plain,markdown,docx,html,jats}
                        When --render, format to use for output file. If not
                        specified, attempt to infer this from filename
                        extension. Otherwise, default to plain.
  --output OUTPUT       Specify a file to write output, otherwise default to
                        stdout.
  --allow-invalid-csl-data
                        Allow CSL Items that do not conform to the JSON
                        Schema. Skips CSL pruning.
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr logging

Pandoc filter

This package creates the pandoc-manubot-cite Pandoc filter, providing access to Manubot's cite-by-ID functionality from within a Pandoc workflow.

Currently, this filter is experimental and subject to breaking changes at any point. At some point in the future, we may migrate entirely from manubot process to pandoc-manubot-cite for citation processing.

usage: pandoc-manubot-cite [-h] [--input [INPUT]] [--output [OUTPUT]]
                           target_format

Pandoc filter for citation by persistent identifier. Filters are command-line
programs that read and write a JSON-encoded abstract syntax tree for Pandoc.
Unless you are debugging, run this filter as part of a pandoc command by
specifying --filter=pandoc-manubot-cite.

positional arguments:
  target_format      output format of the pandoc command, as per Pandoc's --to
                     option

optional arguments:
  -h, --help         show this help message and exit
  --input [INPUT]    path read JSON input (defaults to stdin)
  --output [OUTPUT]  path to write JSON output (defaults to stdout)

Other Pandoc filters exist that do something similar: pandoc-url2cite & pwcite. Currently, pandoc-manubot-cite supports the most types of persistent identifiers. We're interested in creating as much compatibility as possible between these filters and their syntaxes.

Webpage

The manubot webpage command populates a webpage directory with Manubot output files.

usage: manubot webpage [-h] [--checkout [CHECKOUT]] [--version VERSION]
                       [--timestamp] [--no-ots-cache | --ots-cache OTS_CACHE]
                       [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]

Update the webpage directory tree with Manubot output files. This command
should be run from the root directory of a Manubot manuscript that follows the
Rootstock layout, containing `output` and `webpage` directories. HTML and PDF
outputs are copied to the webpage directory, which is structured as static
source files for website hosting.

optional arguments:
  -h, --help            show this help message and exit
  --checkout [CHECKOUT]
                        branch to checkout /v directory contents from. For
                        example, --checkout=upstream/gh-pages. --checkout is
                        equivalent to --checkout=gh-pages. If --checkout is
                        ommitted, no checkout is performed.
  --version VERSION     Used to create webpage/v/{version} directory.
                        Generally a commit hash, tag, or 'local'. When
                        omitted, version defaults to the commit hash on CI
                        builds and 'local' elsewhere.
  --timestamp           timestamp versioned manuscripts in webpage/v using
                        OpenTimestamps. Specify this flag to create timestamps
                        for the current HTML and PDF outputs and upgrade any
                        timestamps from past manuscript versions.
  --no-ots-cache        disable the timestamp cache.
  --ots-cache OTS_CACHE
                        location for the timestamp cache (default:
                        ci/cache/ots).
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level for stderr logging

Development

Environment

Create a development environment using:

conda create --name manubot-dev --channel conda-forge \
  python=3.7 pandoc=2.8
conda activate manubot-dev  # assumes conda >= 4.4
pip install --editable ".[all]"

Commands

Below are some common commands used for development. They assume the working directory is set to the repository's root, and the conda environment is activated.

# run the test suite
pytest

# reformat Python files according to the black style rules (required to pass CI)
black .

# detect any flake8 linting violations
flake8

# regenerate the README codeblocks for --help messages
python manubot/tests/test_readme.py

# generate the docs
portray as_html --overwrite --output_dir=docs

# process the example testing manuscript
manubot process \
  --content-directory=manubot/process/tests/manuscripts/example/content \
  --output-directory=manubot/process/tests/manuscripts/example/output \
  --log-level=DEBUG

Release instructions

This section is only relevant for project maintainers. Travis CI deployments are used to upload releases to PyPI. To create a new release, bump the __version__ in manubot/__init__.py. Then run the following commands:

TAG=v`python setup.py --version`
# Commit updated __version__ info
git add manubot/__init__.py release-notes/$TAG.md
git commit --message="Prepare $TAG release"
git push
# Create & push tag (assuming upstream is the manubot organization remote)
git tag --annotate $TAG --file=release-notes/$TAG.md
git push upstream $TAG

Goals & Acknowledgments

Our goal is to create scholarly infrastructure that encourages open science and assists reproducibility. Accordingly, we hope for the Manubot software and philosophy to be adopted widely, by both academic and commercial entities. As such, Manubot is free/libre and open source software (see LICENSE.md).

We would like to thank the contributors and funders whose support makes this project possible. Specifically, Manubot development has been financially supported by:

• the Alfred P. Sloan Foundation in Grant G-2018-11163 to @dhimmel.
• the Gordon & Betty Moore Foundation (@DDD-Moore) in Grant GBMF4552 to @cgreene.