ga4gh 0.1.0a1

Installing the reference server

Prerequisites:

• Python 2.7,

• Berkeley DB together with include and lib files (version 4.8 or higher),

• Virtualenv (or another python sandboxing tool) is highly recommended.

General installation procedure:

• Install Berkeley DB (version 4.8 or higher) using your system’s preferred package manager, see the wormtable help page for platform-specific details.

• (On MacOS X, make sure the LDFLAGS and CFLAGS environment variables are set to include the lib and include directories for the Berkeley DB install of your choice. The wormtable help page cited above provides more detailed instructions, or see the Appendix to this README for an example install on that platform.)

• Create a python sandbox directory using virtualenv, preferably not inside the ga4gh server directory. For an good introduction to using virtualenv, see the Python Guide page. On some systems, you may need to specify the –no-site-packages option to ensure a clean dependency install. For example, to create a sandbox named testenv:

  $ virtualenv --no-site-packages testenv

• Make the virtualenv sandbox created above active:

  $ source testenv/bin/activate

• cd to the ga4gh server directory, and load the dependencies via pip:

  $ cd [your ga4gh server directory]
  $ pip install -r requirements.txt

• Finally, run the install script, and run nosetests to confirm the install:

  $ python setup.py install
  $ nosetests

A successfull install should result in a clean run of all the tests, resulting in a line of dots followed by OK. If this still isn’t working, you may want to check the Appendix at the end of this README for system-specific installation examples.

Serving variants from a VCF file

Two implementations of the variants API are available that can serve data based on existing VCF files. These backends are based on tabix and wormtable, which is a Python library to handle large scale tabular data. See Wormtable backend for instructions on serving VCF data from the GA4GH API.

Wormtable backend

The wormtable backend allows us to serve variants from an arbitrary VCF file. The VCF file must first be converted to wormtable format using the vcf2wt utility (the wormtable tutorial discusses this process). A subset (1000 rows for each chromosome) of the 1000 Genomes VCF data (20110521 and 20130502 releases) has been prepared and converted to wormtable format and made available here. See Converting 1000G data for more information on converting 1000 genomes data into wormtable format.

To run the server on this example dataset, follow the steps on installing the server, then download and unpack the example data

$ wget http://www.well.ox.ac.uk/~jk/ga4gh-example-data.tar.gz
$ tar -zxvf ga4gh-example-data.tar.gz

An easier way to download and upack the data is to run the following script, which will do these steps for you:

$ python scripts/update_data.py

You can now run the server, telling it to serve variants from the sets in the downloaded datafile:

$ ga4gh_server ga4gh-example-data

To run queries against this server, we can use the ga4gh_client program; for example, here we run the variants/search method over the 1000g_2013.wt variant set, where the reference name is 1 and we only want calls returned for call set ID HG03279:

$ ga4gh_client variants-search http://localhost:8000/v0.5.1 -V 1000g_2013.wt -r 1 -c HG03279 | less -S

We can also query against the variant name; here we return the variant that has variant name rs75454623:

$ ga4gh_client variants-search http://localhost:8000/v0.5.1 -V 1000g_2013.wt -r 1 -n rs75454623  | less -S

$ vcf2wt 1000g_2013-subset.vcf -s schema-1000g_2013.xml -t 1000g_2013

$ wtadmin add 1000g_2013 CHROM+POS
$ wtadmin add 1000g_2013 CHROM+ID

Tabix backend

The tabix backend allows us to serve variants from an arbitrary VCF file. The VCF file must first be indexed with tabix. Many projects, including the 1000 genomes project, release files with tabix indices already precomputed. This backend can serve such datasets without any preprocessing via the command:

$ ga4gh_server DATADIR

where DATADIR is a directory that contains subdirectories of tabix-indexed VCF file(s). There cannot be more than one VCF file in any subdirectory that has data for the same reference contig.

Layout

The code for the project is held in the ga4gh package, which corresponds to the ga4gh directory in the project root. Within this package, the functionality is split between the client, server, protocol and cli modules. The cli module contains the definitions for the ga4gh_client and ga4gh_server programs.

For development purposes, it is useful to be able to run the command line programs directly without installing them. To do this, use the server_dev.py and client_dev.py scripts. (These are just shims to facilitate development, and are not intended to be distributed. The distributed versions of the programs are packaged using the setuptools entry_point key word; see setup.py for details). For example, the run the server command simply run:

$ python server_dev.py
usage: server_dev.py [-h] [--port PORT] [--verbose] {help,wormtable,tabix} ...
server_dev.py: error: too few arguments

Deployment

TODO Give simple instructions for deploying the server on common platforms like Apache and Nginx.

Configuration parameters are specified in the file ga4gh/server/config.py; they can be overridden by setting the absolute path of a file containing new values in the environment variable GA4GH_CONFIGURATION.

Appendix

system specific install examples

MacOS X (with MacPorts):

$ sudo port install db48
$ export CFLAGS=-I/opt/local/include/db48/  LDFLAGS=-L/opt/local/lib/db48/
$ cd [some working directory outside the ga4gh server directory tree]
$ virtualenv --no-site-packages testenv
$ source testenv/bin/activate
$ cd [your ga4gh server directory]
$ pip install -r requirements.txt
$ python setup.py install
$ nosetests

TODO Append examples of installs (using package managers if possible, no dependency installs from source) on the target platform of your choice.