A simple JIT job management framework in Python.
Project description
Custodian is a simple, robust and flexible just-in-time (JIT) job management framework written in Python. Using custodian, you can create wrappers that perform error checking, job management and error recovery. It has a simple plugin framework that allows you to develop specific job management workflows for different applications.
Error recovery is an important aspect of many high-throughput projects that generate data on a large scale. When you are running on the order of hundreds of thousands of jobs, even an error-rate of 1% would mean thousands of errored jobs that would be impossible to deal with on a case-by-case basis.
The specific use case for custodian is for long running jobs, with potentially random errors. For example, there may be a script that takes several days to run on a server, with a 1% chance of some IO error causing the job to fail. Using custodian, one can develop a mechanism to gracefully recover from the error, and potentially restart the job if necessary.
Getting custodian
Stable version
The version at the Python Package Index (PyPI) is always the latest stable release that will be hopefully, be relatively bug-free. The easiest way to install custodian on any system is to use easy_install or pip, as follows:
easy_install custodian
or:
pip install custodian
Some plugins (e.g., vasp management) require additional setup (please see pymatgen’s documentation).
Developmental version
The bleeding edge developmental version is at the custodian’s Github repo. The developmental version is likely to be more buggy, but may contain new features. The Github version include test files as well for complete unit testing. After cloning the source, you can type:
python setup.py install
or to install the package in developmental mode:
python setup.py develop
Requirements
Custodian requires Python 2.7+. There are no other required dependencies.
Optional dependencies
Optional libraries that are required if you need certain features:
pymatgen 2.6.2+: To use the plugin for VASP. Please install using:
pip install pymatgen
For more information, please consult pymatgen’s documentation.
nose - For complete unittesting.
Basic usage
The main class in the workflow is known as Custodian, which manages a series of jobs with a list of error handlers. To use custodian, you need to implement concrete implementation of the abstract base classes custodian.custodian.Job and custodian.custodian.ErrorHandler. An very simple example implementation is given in the custodian_examples.py script in the scripts directory.
Electronic structure calculations
Other specific examples for electronic structure calculations based on the Vienna Ab Initio Simulation Package (VASP) are implemented in the custodian.vasp package. A simple example of a script using Custodian to run a two-relaxation VASP job is as follows:
from custodian.custodian import Custodian
from custodian.vasp.handlers import VaspErrorHandler, UnconvergedErrorHandler
from custodian.vasp.jobs import VaspJob
handlers = [VaspErrorHandler(), UnconvergedErrorHandler(),
PoscarErrorHandler()]
jobs = VaspJob.double_relaxation_run(args.command.split())
c = Custodian(handlers, jobs, max_errors=10)
c.run()The above will gracefully deal with many VASP errors encountered during relaxation. For example, it will correct ISMEAR to 0 if there are insufficient KPOINTS to use ISMEAR = -5.
Using custodian, you can even setup potentially indefinite jobs, e.g. kpoints convergence jobs with a target energy. Please see the converge_kpoints scripts for an example.
How to cite custodian
If you use custodian in your research, especially the VASP component, please consider citing the following work:
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent Chevrier, Kristin A. Persson, Gerbrand Ceder. Python Materials Genomics (pymatgen) : A Robust, Open-Source Python Library for Materials Analysis. Computational Materials Science, 2013, 68, 314–319. doi:10.1016/j.commatsci.2012.10.028
License
Custodian is released under the MIT License. The terms of the license are as follows:
The MIT License (MIT) Copyright (c) 2011-2012 MIT & LBNL Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software") , to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file custodian-0.3.2.tar.gz.
File metadata
- Download URL: custodian-0.3.2.tar.gz
- Upload date:
- Size: 16.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3ebfed393cac80e6c35058ae00f7e050d3514c95041ba83e60a795502cb13b72 |
|
| MD5 | ffacdd49e39b584bfa1812e9aaacbb8a |
|
| BLAKE2b-256 | 1cfc475fcf9f7c552296893ebe8678c5ed6e4855b86446c9bf35f403417b55a7 |
File details
Details for the file custodian-0.3.2-py2.7.egg.
File metadata
- Download URL: custodian-0.3.2-py2.7.egg
- Upload date:
- Size: 44.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b09351b41c8b2a319451137a03d7d51713eca91acc31d908c842fa6cb8c9b569 |
|
| MD5 | 7a53f209d43d3c34ac032531047d3da2 |
|
| BLAKE2b-256 | e27fd79332f7d3e227d55465a756e0ffdcffb9cc4a9fd02e79231edebc6c4ca3 |
