Skip to main content

Biohackathon sequence uploader

Project description

COVID-19 PubSeq: Public Sequence uploader

Join the chat at https://gitter.im/arvados/pubseq

This repository provides a sequence uploader for the COVID-19 Virtual Biohackathon's Public Sequence Resource project. There are two versions, one that runs on the command line and another that acts as web interface. You can use it to upload the genomes of SARS-CoV-2 samples to make them publicly and freely available to other researchers. For more information see the paper.

alt text

To get started, first install the uploader, and use the bh20-seq-uploader command to upload your data.

Installation

There are several ways to install the uploader. The most portable is with a virtualenv.

Installation with virtualenv

  1. Prepare your system. You need to make sure you have Python, and the ability to install modules such as pycurl and pyopenssl. On Ubuntu 18.04, you can run:
sudo apt update
sudo apt install -y virtualenv git libcurl4-openssl-dev build-essential python3-dev libssl-dev
  1. Create and enter your virtualenv. Go to some memorable directory and make and enter a virtualenv:
virtualenv --python python3 venv
. venv/bin/activate

Note that you will need to repeat the . venv/bin/activate step from this directory to enter your virtualenv whenever you want to use the installed tool.

  1. Install the tool. Once in your virtualenv, install this project:

Install from PyPi:

pip3 bh20-seq-uploader

Install from git:

pip3 install git+https://github.com/arvados/bh20-seq-resource.git@master
  1. Test the tool. Try running:
bh20-seq-uploader --help

It should print some instructions about how to use the uploader.

Make sure you are in your virtualenv whenever you run the tool! If you ever can't run the tool, and your prompt doesn't say (venv), try going to the directory where you put the virtualenv and running . venv/bin/activate. It only works for the current terminal window; you will need to run it again if you open a new terminal.

Installation with pip3 --user

If you don't want to have to enter a virtualenv every time you use the uploader, you can use the --user feature of pip3 to install the tool for your user.

  1. Prepare your system. Just as for the virtualenv method, you need to install some dependencies. On Ubuntu 18.04, you can run:
sudo apt update
sudo apt install -y virtualenv git libcurl4-openssl-dev build-essential python3-dev libssl-dev
  1. Install the tool. You can run:
pip3 install --user git+https://github.com/arvados/bh20-seq-resource.git@master
  1. Make sure the tool is on your PATH. The pip3 command will install the uploader in .local/bin inside your home directory. Your shell may not know to look for commands there by default. To fix this for the terminal you currently have open, run:
export PATH=$PATH:$HOME/.local/bin

To make this change permanent, assuming your shell is Bash, run:

echo 'export PATH=$PATH:$HOME/.local/bin' >>~/.bashrc
  1. Test the tool. Try running:
bh20-seq-uploader --help

It should print some instructions about how to use the uploader.

Installation from Source for Development

If you plan to contribute to the project, you may want to install an editable copy from source. With this method, changes to the source code are automatically reflected in the installed copy of the tool.

  1. Prepare your system. On Ubuntu 18.04, you can run:
sudo apt update
sudo apt install -y virtualenv git libcurl4-openssl-dev build-essential python3-dev libssl-dev
  1. Clone and enter the repository. You can run:
git clone https://github.com/arvados/bh20-seq-resource.git
cd bh20-seq-resource
  1. Create and enter a virtualenv. Go to some memorable directory and make and enter a virtualenv:
virtualenv --python python3 venv
. venv/bin/activate

Note that you will need to repeat the . venv/bin/activate step from this directory to enter your virtualenv whenever you want to use the installed tool.

  1. Install the checked-out repository in editable mode. Once in your virtualenv, install with this special pip command:
pip3 install -e .
  1. Test the tool. Try running:
bh20-seq-uploader --help

It should print some instructions about how to use the uploader.

Installation with GNU Guix

For running/developing the uploader with GNU Guix see INSTALL.md

Usage

Run the uploader with a FASTA or FASTQ file and accompanying metadata file in JSON or YAML:

bh20-seq-uploader example/sequence.fasta example/metadata.yaml

Workflow for Generating a Pangenome

All these uploaded sequences are being fed into a workflow to generate a pangenome for the virus. You can replicate this workflow yourself.

An example is to get your SARS-CoV-2 sequences from GenBank in seqs.fa, and then run a series of commands

minimap2 -cx asm20 -X seqs.fa seqs.fa >seqs.paf
seqwish -s seqs.fa -p seqs.paf -g seqs.gfa
odgi build -g seqs.gfa -s -o seqs.odgi
odgi viz -i seqs.odgi -o seqs.png -x 4000 -y 500 -R -P 5

Here we convert such a pipeline into the Common Workflow Language (CWL) and sources can be found here.

For more information on building pangenome models, see this wiki page.

Web Interface

This project comes with a simple web server that lets you use the sequence uploader from a browser. It will work as long as you install the packager with the web extra.

To run it locally:

virtualenv --python python3 venv
. venv/bin/activate
pip install -e ".[web]"
env FLASK_APP=bh20simplewebuploader/main.py flask run

Then visit http://127.0.0.1:5000/.

Production

For production deployment, you can use gunicorn:

pip3 install gunicorn
gunicorn bh20simplewebuploader.main:app

This runs on http://127.0.0.1:8000/ by default, but can be adjusted with various gunicorn options.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bh20-seq-uploader-1.0.20200625072247.tar.gz (32.6 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file bh20-seq-uploader-1.0.20200625072247.tar.gz.

File metadata

  • Download URL: bh20-seq-uploader-1.0.20200625072247.tar.gz
  • Upload date:
  • Size: 32.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.3

File hashes

Hashes for bh20-seq-uploader-1.0.20200625072247.tar.gz
Algorithm Hash digest
SHA256 139015ed0dbc0b337ac3e3e17b1ca807099d2e7d01245ff77068fcb48b90b026
MD5 280acd1e1895c0fd94a41bd45c0c2d12
BLAKE2b-256 891b5772ec5fa6e50356363ee3978df49df8d5e0b7e4f4879dba7963919e4de4

See more details on using hashes here.

File details

Details for the file bh20_seq_uploader-1.0.20200625072247-py3-none-any.whl.

File metadata

  • Download URL: bh20_seq_uploader-1.0.20200625072247-py3-none-any.whl
  • Upload date:
  • Size: 38.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.3

File hashes

Hashes for bh20_seq_uploader-1.0.20200625072247-py3-none-any.whl
Algorithm Hash digest
SHA256 19c997810f8ba3db6c3604b5503f3ff8b8f542dc1c1fecbf5a990db2a627715b
MD5 14e184ea439e0a353730f3a2d58a815e
BLAKE2b-256 f5e15335e06bde12a0afb7d4f320de3435d206c02fe5e365b9f4ce4dbcebc5d6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page