Skip to main content

Data Biosphere Data Store Command Line Interface

Project description

This repository is a pip-installable command line interface (CLI) and Python library (API) for interacting with the Data Biosphere Data Storage System (DSS), also called the data store.

Currently the dbio package supports interaction with the data store for uploading, downloading, and fetching information about data in the data store.

The Data Biosphere CLI is compatible with Python versions 3.5+.

Installation

pip install dbio.

Usage

Documentation on readthedocs.io:

To see the list of commands you can use, type dbio --help.

Configuration management

The Data Biosphere CLI supports ingesting configuration from a configurable array of sources. Each source is a JSON file. Configuration sources that follow the first source update the configuration using recursive dictionary merging. Sources are enumerated in the following order (i.e., in order of increasing priority):

  • Site-wide configuration source, /etc/dbio/config.json

  • User configuration source, ~/.config/dbio/config.json

  • Any sources listed in the colon-delimited variable DBIO_CONFIG_FILE

  • Command line options

Array merge operators: When loading a chain of configuration sources, the Data Biosphere CLI uses recursive dictionary merging to combine the sources. Additionally, when the original config value is a list, the package supports array manipulation operators, which let you extend and modify arrays defined in underlying configurations. See https://github.com/kislyuk/tweak#array-merge-operators for a list of these operators.

Service to Service Authorization

Google service credentials must be whitelisted before they will authenticate with the Data Biosphere CLI.

Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of your Google service credentials file to authenticate.

One can also use: dbio dss login.

See Google service credentials for more information about service accounts. Use the Google Cloud IAM web console to manage service accounts.

Development

To develop on the CLI, first run pip install -r requirements-dev.txt. You can install your locally modified copy of the dbio package by running make install in the repository root directory.

To use the command line interface with a local or test DSS, first run dbio (or scripts/dbio if you want to use the package in-place from the repository root directory). This will create the file ~/.config/dbio/config.json, which you can modify to update the value of DSSClient.swagger_url to point to the URL of the Swagger definition served by your DSS deployment. Lastly, the CLI enforces HTTPS connection to the DSS API. If you are connecting to a local DSS, make this change in dbio/util/__init__.py in the SwaggerClient object:

scheme = "http"

To use the Python interface with a local or test DSS, pass the URL of the Swagger definition to the DSSClient constructor via the swagger_url parameter:

client = DSSClient(swagger_url="https://dss.example.com/v1/swagger.json")

You can also layer a minimal config file on top of the default config.json using the DBIO_CONFIG_FILE environment variable, for example:

export SWAGGER_URL="https://dss.dev.ucsc-cgp-redwood.org/v1/swagger.json"
jq -n .DSSClient.swagger_url=env.SWAGGER_URL > ~/.config/dbio/config.staging.json
export DBIO_CONFIG_FILE=~/.config/dbio/config.staging.json

Testing

Before you run tests, first run dbio dss login. This will open a browser where you can log in to authenticate with Google. Use an email address from one of the whitelisted domains (in DSS_SUBSCRIPTION_AUTHORIZED_DOMAINS_ARRAY from here).

Then make test.

Primary CI testing is through Travis CI on the Gitlab toilspark instance.

Bugs

Please report bugs, issues, feature requests, etc. in the HumanCellAtlas/dcp-cli repository on GitHub.

Security Policy

Please email reports about any security related issues you find to team-redwood@ucsc.edu. Use a descriptive subject line for your report email. In addition, please include the following information along with your report:

  • Your name and affiliation (if any).

  • A description of the technical details of the vulnerabilities, to help us reproduce your findings.

  • An explanation of who can exploit this vulnerability, and what they gain when doing so (an attack scenario).

  • Whether this vulnerability is public or known to third parties. If so, please provide details.

License

Licensed under the terms of the MIT License.

https://img.shields.io/travis/DataBiosphere/data-store-cli.svg?branch=master https://codecov.io/github/DataBiosphere/data-store-cli/coverage.svg?branch=master https://img.shields.io/pypi/v/dbio.svg https://img.shields.io/pypi/l/dbio.svg https://readthedocs.org/projects/dbio-cli/badge/?version=latest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbio-cli-1.0.0.tar.gz (61.6 kB view details)

Uploaded Source

Built Distribution

dbio_cli-1.0.0-py2.py3-none-any.whl (61.0 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file dbio-cli-1.0.0.tar.gz.

File metadata

  • Download URL: dbio-cli-1.0.0.tar.gz
  • Upload date:
  • Size: 61.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.6.3

File hashes

Hashes for dbio-cli-1.0.0.tar.gz
Algorithm Hash digest
SHA256 bcf708b580dfad8bc39ca0389efbb175436f35c0eaa08faa98c42c7b52778a4c
MD5 26133506f0baecc14612435c56957e9d
BLAKE2b-256 ac20f65b6af4e9a075c0446ed3255b4b5056675c3727833bedb95c59c9225e69

See more details on using hashes here.

File details

Details for the file dbio_cli-1.0.0-py2.py3-none-any.whl.

File metadata

  • Download URL: dbio_cli-1.0.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 61.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.6.3

File hashes

Hashes for dbio_cli-1.0.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 e4d1d1d33c89f2f27f9bed3d97c73f9d3246296290be63df02d6c0f28b799324
MD5 226e9d67c9a92cfcbee7299543eaa431
BLAKE2b-256 3fd3f544255d4f3b3df6e9f2360e7782373246f6718f7962e4d9bb855720dff2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page