Distributed job execution application built for Galaxy (http://galaxyproject.org/).
Project description
This project is a Python server application that allows a Galaxy server to run jobs on remote systems (including Windows) without requiring a shared mounted file systems. Unlike traditional Galaxy job runners - input files, scripts, and config files may be transferred to the remote system, the job is executed, and the results are transfered back to the Galaxy server - eliminating the need for a shared file system.
Full documentation for the project can be found on Read The Docs.
Configuring Galaxy
Galaxy job runners are configured in Galaxy’s job_conf.xml file. Some small examples of how to configure this can be found here, but be sure to checkout job_conf.xml.sample_advanced in your Galaxy code base or on Github for complete information.
Quickstart
Full details on different ways to install Pulsar can be found in the install section of the documentaiton, but if your machine has the proper Python dependencies available it can be quickly download and a test job run with.
mkdir pulsar cd pulsar virtualenv venv . venv/bin/activate # .venv\Scripts\activate if Windows pip install pulsar-app pulsar-config pulsar --daemon # just pulsar if Windows pulsar-check # runs a test job
The configuration documentation has many details on securing your Pulsar server and enabling advanced features such as cluster integration and message queue communication.
Development and Testing
The recommended approach to setting up a development environment for Pulsar on Linux or Mac OS X is rougly as follows:
git clone https://github.com/galaxyproject/pulsar cd pulsar virtualenv .venv . .venv/bin/activate # .venv\Scripts\activate if Windows pip install -r requirements.txt pip install -r dev-requirements.txt
This project is distributed with unit and integration tests (many of which will not run under Windows), the following command will install the needed python components to run these tests. The following command will then run these tests:
make tests
The following command will then produce a coverage report corresponding to this test and place it in the coverage_html_report subdirectory of this project.:
coverage html
Checkout the Contributing documentation for many more details on developing and contributing to Pulsar.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
History
0.8.0 (2017-09-21)
Support new features in Galaxy job running/scripting so that Pulsar respects $GALAXY_VIRTUAL_ENV and $PRESERVE_GALAXY_ENVIRONMENT. Fix remote metadata in cases where the tool environment changes the python on $PATH. Pull Request 137
Precreate Galaxy tool outputs on the remote before executing (fixes a bug related to missing output files on stage out). Pull Request 141
Support the remote_transfer file action without setting the jobs_directory destination param Pull Request 136
Fix invalid character in job managers documentation (thanks to @mapa17). Pull Request 130
Fix conda_auto_* option resolution and include a sample dependency_resolvers_conf.xml (thanks to @mapa17). Pull Request 132
Fix tox/Travis tests. Pull Request 138, Pull Request 139, Pull Request 140
Fix a bug with AMQP acknowledgement. Pull Request 143
0.7.4 (2017-02-07)
0.7.3 (2016-10-31)
Fix “AttributeError” when submitting a job as a real user. Pull Request 124, Issue 123
0.7.2 (2016-08-31)
Fix bug causing loops on in response to preprocessing error conditions.
0.7.1 (2016-08-29)
Do a release to circumvent a tool version logic error in Galaxy ( released Galaxy versions think 0.7.0 < 0.7.0.dev3).
0.7.0 (2016-08-26)
Update Makefile to allow release pulsar as an application and a library for Galaxy at the same time.
Small update to test scripts for TravisCI changes.
Improvements for embedded Galaxy runner. (TODO: fill this out)
Remove support for Python 2.6. 60bf962
Update docs to describe project goverance and reuse Galaxy’s Code of Conduct. 7e23d43, dc47140
Updated cluster slots detection for SLURM from Galaxy. cadfc5a
Various changes to allow usage within Galaxy as a library. ce9d4f9
Various changes to allow embedded Pulsar managers within Galaxy. ce9d4f9, d262323, 8f7c04a
Introduce a separate working and metadata directory as required for Galaxy 16.04 that requires this separation. 6f4328e
Add Tox target for Python 2.7 unit testing. d7c524e
Add Makefile command for setup.py develop. fd82d00
0.6.1 (2015-12-23)
Tweak release process that left 0.6.0 with an incorrect PyPI description page.
0.6.0 (2015-12-23)
Pulsar now depends on the new galaxy-lib Python package instead of manually synchronizing Python files across Pulsar and Galaxy.
Numerous build and testing improvements.
Fixed a documentation bug in the code (thanks to @erasche). e8814ae
Remove galaxy.eggs stuff from Pulsar client (thanks to @natefoo). 00197f2
Add new logo to README (thanks to @martenson). abbba40
Implement an optional awknowledgement system on top of the message queue system (thanks to @natefoo). Pull Request 82 431088c
Documentation fixes thanks to @remimarenco. Pull Request 78, Pull Request 80
Fix project script bug introduced this cycle (thanks to @nsoranzo). 140a069
Fix config.py on Windows (thanks to @ssorgatem). Pull Request 84
Add a job manager for XSEDE jobs (thanks to @natefoo). 1017bc5
Fix pip dependency installation (thanks to @afgane) Pull Request 73
0.5.0 (2015-05-08)
0.4.0 (2015-04-20)
Python 3 support. Pull Request 62
Fix bug encountered when running pulsar-main and pulsar-config commands as scripts. 9d43ae0
Add pulsar-run script for issues commands against a Pulsar server (experimental). 3cc7f74
0.3.0 (2015-04-12)
Changed the name of project to Pulsar, moved to Github.
New RESTful web services interface.
SCP and Rsync file staging options added by Eric Rasche. Pull Request
Allow YAML based configuration.
Support for more traditional pip/setup.py-style installs.
Dozens of smaller bugfixes and documentation updates.
0.2.0
Last version named the LWR - found on BitBucket.
Still supported in Galaxy as of 15.03 the release.
Introduced support for submitting to various queueing systems, operation as a Mesos framework, Docker support, and various other advanced deployment options.
Message queue support.
Framework for configurable file actions introduced.
0.1.0
Simple support for running jobs managed by the Python LWR web process.
0.0.1
See the original announcement and initial commit.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file pulsar-galaxy-lib-0.8.0.tar.gz.
File metadata
- Download URL: pulsar-galaxy-lib-0.8.0.tar.gz
- Upload date:
- Size: 134.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8fdc4a26874bf9235b2de6544a8fa06c4b65b196f6d8cae6f8fac52e4c43d5d7 |
|
| MD5 | bf64c6b8d23c4c28483be289c7510fe1 |
|
| BLAKE2b-256 | d15b98f9e49d61d008b7f9143842080a0cd13298a2a5369af79413bf96c82be1 |
File details
Details for the file pulsar_galaxy_lib-0.8.0-py2-none-any.whl.
File metadata
- Download URL: pulsar_galaxy_lib-0.8.0-py2-none-any.whl
- Upload date:
- Size: 163.9 kB
- Tags: Python 2
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7dbb28f2c9d56147d00b72a9a05dbb99479c876e11c7836c04cb0e7bd99b2b58 |
|
| MD5 | 462c366845f016decbf1af7ed4c18cc6 |
|
| BLAKE2b-256 | 0884551d53b4750bd46026e755cceb2a0b8e69ab661636fc7f56b164ad73df28 |
