Skip to main content

Distributed job execution application built for Galaxy (http://galaxyproject.org/).

Project description

Pulsar Logo
https://readthedocs.org/projects/pip/badge/?version=latest https://travis-ci.org/galaxyproject/pulsar.png?branch=master https://coveralls.io/repos/galaxyproject/pulsar/badge.png?branch=master

This project is a Python server application that allows a Galaxy server to run jobs on remote systems (including Windows) without requiring a shared mounted file systems. Unlike traditional Galaxy job runners - input files, scripts, and config files may be transferred to the remote system, the job is executed, and the results are transfered back to the Galaxy server - eliminating the need for a shared file system.

Full documentation for the project can be found on Read The Docs.

Configuring Galaxy

Galaxy job runners are configured in Galaxy’s job_conf.xml file. Some small examples of how to configure this can be found here, but be sure to checkout job_conf.xml.sample_advanced in your Galaxy code base or on Github for complete information.

Quickstart

Full details on different ways to install Pulsar can be found in the install section of the documentaiton, but if your machine has the proper Python dependencies available it can be quickly download and a test job run with.

mkdir pulsar
cd pulsar
virtualenv venv
. venv/bin/activate # .venv\Scripts\activate if Windows
pip install pulsar-app
pulsar-config
pulsar --daemon # just pulsar if Windows
pulsar-check # runs a test job

The configuration documentation has many details on securing your Pulsar server and enabling advanced features such as cluster integration and message queue communication.

Development and Testing

The recommended approach to setting up a development environment for Pulsar on Linux or Mac OS X is rougly as follows:

git clone https://github.com/galaxyproject/pulsar
cd pulsar
virtualenv .venv
. .venv/bin/activate # .venv\Scripts\activate if Windows
pip install -r requirements.txt
pip install -r dev-requirements.txt

This project is distributed with unit and integration tests (many of which will not run under Windows), the following command will install the needed python components to run these tests. The following command will then run these tests:

make tests

The following command will then produce a coverage report corresponding to this test and place it in the coverage_html_report subdirectory of this project.:

coverage html

Checkout the Contributing documentation for many more details on developing and contributing to Pulsar.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

History

0.7.0 (2016-08-26)

  • Update Makefile to allow release pulsar as an application and a library for Galaxy at the same time.

  • Small update to test scripts for TravisCI changes.

  • Improvements for embedded Galaxy runner. (TODO: fill this out)

  • Remove support for Python 2.6. 60bf962

  • Update docs to describe project goverance and reuse Galaxy’s Code of Conduct. 7e23d43, dc47140

  • Updated cluster slots detection for SLURM from Galaxy. cadfc5a

  • Various changes to allow usage within Galaxy as a library. ce9d4f9

  • Various changes to allow embedded Pulsar managers within Galaxy. ce9d4f9, d262323, 8f7c04a

  • Introduce a separate working and metadata directory as required for Galaxy 16.04 that requires this separation. 6f4328e

  • Improve logging and comments. 38953f3, a985107, ad33cb9

  • Add Tox target for Python 2.7 unit testing. d7c524e

  • Add Makefile command for setup.py develop. fd82d00

0.6.1 (2015-12-23)

  • Tweak release process that left 0.6.0 with an incorrect PyPI description page.

0.6.0 (2015-12-23)

  • Pulsar now depends on the new galaxy-lib Python package instead of manually synchronizing Python files across Pulsar and Galaxy.

  • Numerous build and testing improvements.

  • Fixed a documentation bug in the code (thanks to @erasche). e8814ae

  • Remove galaxy.eggs stuff from Pulsar client (thanks to @natefoo). 00197f2

  • Add new logo to README (thanks to @martenson). abbba40

  • Implement an optional awknowledgement system on top of the message queue system (thanks to @natefoo). Pull Request 82 431088c

  • Documentation fixes thanks to @remimarenco. Pull Request 78, Pull Request 80

  • Fix project script bug introduced this cycle (thanks to @nsoranzo). 140a069

  • Fix config.py on Windows (thanks to @ssorgatem). Pull Request 84

  • Add a job manager for XSEDE jobs (thanks to @natefoo). 1017bc5

  • Fix pip dependency installation (thanks to @afgane) Pull Request 73

0.5.0 (2015-05-08)

  • Allow cURL downloader to resume transfers during staging in (thanks to @natefoo). 0c61bd9

  • Fix to cURL downloaders status code handling (thanks to @natefoo). 86f95ce

  • Fix non-wheel installs from PyPI. Issue 72

  • Fix mesos imports for newer versions of mesos (thanks to @kellrott). fe3e919

  • More, better logging. 2b3942d, fa2b6dc

0.4.0 (2015-04-20)

  • Python 3 support. Pull Request 62

  • Fix bug encountered when running pulsar-main and pulsar-config commands as scripts. 9d43ae0

  • Add pulsar-run script for issues commands against a Pulsar server (experimental). 3cc7f74

0.3.0 (2015-04-12)

  • Changed the name of project to Pulsar, moved to Github.

  • New RESTful web services interface.

  • SCP and Rsync file staging options added by Eric Rasche. Pull Request

  • Allow YAML based configuration.

  • Support for more traditional pip/setup.py-style installs.

  • Dozens of smaller bugfixes and documentation updates.

0.2.0

  • Last version named the LWR - found on BitBucket.

  • Still supported in Galaxy as of 15.03 the release.

  • Introduced support for submitting to various queueing systems, operation as a Mesos framework, Docker support, and various other advanced deployment options.

  • Message queue support.

  • Framework for configurable file actions introduced.

0.1.0

0.0.1

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pulsar-galaxy-lib-0.7.0.tar.gz (136.6 kB view details)

Uploaded Source

Built Distributions

pulsar_galaxy_lib-0.7.0-py2.7.egg (369.6 kB view details)

Uploaded Source

pulsar_galaxy_lib-0.7.0-py2-none-any.whl (160.8 kB view details)

Uploaded Python 2

File details

Details for the file pulsar-galaxy-lib-0.7.0.tar.gz.

File metadata

File hashes

Hashes for pulsar-galaxy-lib-0.7.0.tar.gz
Algorithm Hash digest
SHA256 58455b80285f7d01a5120c6decc0d776cfacc053daf1cc3fdb19f5d15cce3468
MD5 a77ca3de3e587356fc64e3c6916e7fc5
BLAKE2b-256 a6e49815a09a4a3b667b5f7880bcd5acbe27d1d2828c4dd23b34109f66838b89

See more details on using hashes here.

File details

Details for the file pulsar_galaxy_lib-0.7.0-py2.7.egg.

File metadata

File hashes

Hashes for pulsar_galaxy_lib-0.7.0-py2.7.egg
Algorithm Hash digest
SHA256 91723b85131872709b63dd13d97a22de97fe0c56431fc3511c356f13f7bd0aa1
MD5 5a5bbf0ec3407cb61a9f227cccaab6a8
BLAKE2b-256 261a6736965878c561b99cb078baeb43a708a1fcf23241dad0641af6e50d9cc9

See more details on using hashes here.

File details

Details for the file pulsar_galaxy_lib-0.7.0-py2-none-any.whl.

File metadata

File hashes

Hashes for pulsar_galaxy_lib-0.7.0-py2-none-any.whl
Algorithm Hash digest
SHA256 41014b085a26c3990dda5ef49ed3b06348e69a9eecce739cfb8abef4f6f616a6
MD5 080a0a8230a5be020f0f31605842956f
BLAKE2b-256 a0dd90159513587bde8edd66c3f6495868a6c0b3cc29d612c64b59762c4b4940

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page