Skip to main content

Simple HPC queuing system adapter for Python on based jinja templates to automate the submission script creation.

Project description

pysqa

Unittests Documentation Status Coverage Status Binder

High-performance computing (HPC) does not have to be hard. In this context the aim of the Python Simple Queuing System Adapter (pysqa) is to simplify the submission of tasks from python to HPC clusters as easy as starting another subprocess locally. This is achieved based on the assumption that even though modern HPC queuing systems offer a wide range of different configuration options, most users submit the majority of their jobs with very similar parameters.

Therefore, in pysqa users define submission script templates once and reuse them to submit many different tasks and workflows afterwards. These templates are defined in the jinja2 template language, so current submission scripts can be easily converted to templates. In addition, to the submission of new tasks to HPC queuing systems, pysqa also allows the users to track the progress of their tasks, delete them or enable reservations using the built-in functionality of the queuing system. Finally, pysqa enables remote connections to HPC clusters using SSH including support for two factor authentication via pyauthenticator, this allows the users to submit task from a python process on their local workstation to remote HPC clusters.

All this functionality is available from both the Python interface as well as the command line interface.

Features

The core feature of pysqa is the communication to HPC queuing systems including (Flux, LFS, MOAB, SGE, SLURM and TORQUE). This includes:

  • QueueAdapter().submit_job() - Submission of new tasks to the queuing system.
  • QueueAdapter().get_queue_status() - List of calculation currently waiting or running on the queuing system.
  • QueueAdapter().delete_job() - Deleting calculation which are currently waiting or running on the queuing system.
  • QueueAdapter().queue_list - List of available queue templates created by the user.
  • QueueAdapter().config - Templates to a specific number of cores, run time or other computing resources. With integrated checks if a given submitted task follows these restrictions.

In addition to these core features, pysqa is continuously extended to support more use cases for a larger group of users. These new features include the support for remote queuing systems:

  • Remote connection via the secure shell protocol (SSH) to access remote HPC clusters.
  • Transfer of files to and from remote HPC clusters, based on a predefined mapping of the remote file system into the local file system.
  • Support for both individual connections as well as continuous connections depending on the network availability.

Finally, there is current work in progress to support a combination of multiple local and remote queuing systems from within pysqa, which are represented to the user as a single resource.

Documentation

License

pysqa is released under the BSD license . It is a spin-off of the pyiron project therefore if you use pysqa for calculation which result in a scientific publication, please cite:

@article{pyiron-paper,
  title = {pyiron: An integrated development environment for computational materials science},
  journal = {Computational Materials Science},
  volume = {163},
  pages = {24 - 36},
  year = {2019},
  issn = {0927-0256},
  doi = {https://doi.org/10.1016/j.commatsci.2018.07.043},
  url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786},
  author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and Jörg Neugebauer},
  keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols},
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pysqa-0.2.2.tar.gz (34.6 kB view details)

Uploaded Source

Built Distribution

pysqa-0.2.2-py3-none-any.whl (42.7 kB view details)

Uploaded Python 3

File details

Details for the file pysqa-0.2.2.tar.gz.

File metadata

  • Download URL: pysqa-0.2.2.tar.gz
  • Upload date:
  • Size: 34.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for pysqa-0.2.2.tar.gz
Algorithm Hash digest
SHA256 b09bcb9725c7d2fdd7c61b820c8ab2947716b1b58dbc7d4cbba9223031c60231
MD5 e6509efebb1d9c2df5e27a4f4770c247
BLAKE2b-256 8f363c84eb365e5e7ab057f115ee44e679d3514a5490600c91f0ac3cc6b14e4c

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysqa-0.2.2.tar.gz:

Publisher: deploy.yml on pyiron/pysqa

Attestations:

File details

Details for the file pysqa-0.2.2-py3-none-any.whl.

File metadata

  • Download URL: pysqa-0.2.2-py3-none-any.whl
  • Upload date:
  • Size: 42.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for pysqa-0.2.2-py3-none-any.whl
Algorithm Hash digest
SHA256 ca8d44563d93b302bb3b5dfe754b365a6ae0e0bc905ee80f39993f8c71f1a7f8
MD5 4b7487e07fdda377b9f2fe7b9bd8333c
BLAKE2b-256 121a6fd1cd9c7accdfc0fa05425166e0279b8fa2b8114e8d40ace7b13f12eb31

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysqa-0.2.2-py3-none-any.whl:

Publisher: deploy.yml on pyiron/pysqa

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page