Execution helpers for simplified usage of subprocess and ssh.
Project description
exec-helpers
Execution helpers for simplified usage of subprocess and ssh. Why another subprocess wrapper and why no clear paramiko?
Historically paramiko offers good ssh client, but with specific limitations: you can call command with timeout, but without receiving return code, or call command and wait for return code, but without timeout processing.
In the most cases, we are need just simple SSH client with comfortable API for calls, calls via SSH proxy and checking return code/stderr. This library offers this functionality with deadlock free polling and friendly result objects (with inline decoding of XML Element tree, YAML, JSON, binary or just strings). In addition this library offers the same API for subprocess calls, but with specific limitation: no parallel calls (for protection from race conditions).
Pros:
STDOUT and STDERR polling during command execution - no deadlocks.
The same API for subprocess and ssh.
Free software: Apache license
Open Source: https://github.com/python-useful-helpers/exec-helpers
PyPI packaged: https://pypi-hypernode.com/pypi/exec-helpers
Self-documented code: docstrings with types in comments
Tested: see badges on top
Support multiple Python versions:
Python 3.6 Python 3.7 Python 3.8
This package includes:
SSHClient - historically the first one helper, which used for SSH connections. Several API calls for sFTP also presents.
SSHAuth - class for credentials storage. SSHClient does not store credentials as-is, but uses SSHAuth for it. Objects of this class can be copied between ssh connection objects, also it used for execute_through_host.
Subprocess - subprocess.Popen wrapper with timeouts, polling and almost the same API, as SSHClient (except specific flags, like cwd for subprocess and get_tty for ssh).
async_api.Subprocess - the same, as Subprocess helper, but works with asyncio. .. note:: for Windows ProactorEventLoop or another non-standard event loop should be used!
ExecResult - class for execution results storage. Contains exit code, stdout, stderr and getters for decoding as JSON, YAML, XML (and LXML) element tree, string, bytearray and brief strings (up to 7 lines).
ExitCodes - enumerator for standard Linux exit codes. BASH return codes (produced from signal codes) also available.
Installation
Standard: pip install exec-helpers Extras:
yaml - install PyYaml for yaml decoding (PyYAML is main decoder, ruamel.YAML also supported as fallback.)
xml - install defusedxml for safe XML parsing to xml.etree.ElementTree.Element.
lxml - install lxml for advanced XML parsing. Can be unsafe.
ALL_FORMATS (all-formats) - install all parsers. When new parsers will be added, it will ne also supported.
Usage
SSHClient
Basic initialization of SSHClient can be done without construction of specific objects:
client = exec_helpers.SSHClient(host, username="username", password="password")If ssh agent is running - keys will be collected by paramiko automatically, but if keys are in specific location - it should be loaded manually and provided as iterable object of paramiko.RSAKey.
For advanced cases or re-use of credentials, SSHAuth object should be used. It can be collected from connection object via property auth.
Creation from scratch:
auth = exec_helpers.SSHAuth(
username='username', # type: typing.Optional[str]
password='password', # type: typing.Optional[str]
key=None, # type: typing.Optional[paramiko.RSAKey]
keys=None, # type: typing.Optional[typing.Iterable[paramiko.RSAKey]],
key_filename=None, # type: typing.Union[typing.List[str], None]
passphrase=None, # type: typing.Optional[str]
)Key is a main connection key (always tried first) and keys are alternate keys. Key filename is a filename or list of filenames with keys, which should be loaded. Passphrase is an alternate password for keys, if it differs from main password. If main key now correct for username - alternate keys tried, if correct key found - it became main. If no working key - password is used and None is set as main key.
Context manager is available, connection is closed and lock is released on exit from context.
Subprocess
Context manager is available, subprocess is killed and lock is released on exit from context.
Base methods
Main methods are execute, check_call and check_stderr for simple executing, executing and checking return code and executing, checking return code and checking for empty stderr output. This methods are almost the same for SSHClient and Subprocess, except specific flags.
result: ExecResult = helper.execute(
command, # type: str
verbose=False, # type: bool
timeout=1 * 60 * 60, # type: typing.Union[int, float, None]
# Keyword only:
log_mask_re=None, # type: typing.Optional[str]
stdin=None, # type: typing.Union[bytes, str, bytearray, None]
open_stdout=True, # type: bool
open_stderr=True, # type: bool
**kwargs
)result: ExecResult = helper.check_call(
command, # type: str
verbose=False, # type: bool
timeout=1 * 60 * 60, # type: type: typing.Union[int, float, None]
error_info=None, # type: typing.Optional[str]
expected=(0,), # type: typing.Iterable[typing.Union[int, ExitCodes]]
raise_on_err=True, # type: bool
# Keyword only:
log_mask_re=None, # type: typing.Optional[str]
stdin=None, # type: typing.Union[bytes, str, bytearray, None]
open_stdout=True, # type: bool
open_stderr=True, # type: bool
exception_class=CalledProcessError, # typing.Type[CalledProcessError]
**kwargs
)result: ExecResult = helper.check_stderr(
command, # type: str
verbose=False, # type: bool
timeout=1 * 60 * 60, # type: type: typing.Union[int, float, None]
error_info=None, # type: typing.Optional[str]
raise_on_err=True, # type: bool
# Keyword only:
expected=(0,), # typing.Iterable[typing.Union[int, ExitCodes]]
log_mask_re=None, # type: typing.Optional[str]
stdin=None, # type: typing.Union[bytes, str, bytearray, None]
open_stdout=True, # type: bool
open_stderr=True, # type: bool
exception_class=CalledProcessError, # typing.Type[CalledProcessError]
)result: ExecResult = helper( # Lazy way: instances are callable and uses `execute`.
command, # type: str
verbose=False, # type: bool
timeout=1 * 60 * 60, # type: typing.Union[int, float, None]
# Keyword only:
log_mask_re=None, # type: typing.Optional[str]
stdin=None, # type: typing.Union[bytes, str, bytearray, None]
open_stdout=True, # type: bool
open_stderr=True, # type: bool
**kwargs
)If no STDOUT or STDERR required, it is possible to disable this FIFO pipes via **kwargs with flags open_stdout=False and open_stderr=False.
The next command level uses lower level and kwargs are forwarded, so expected exit codes are forwarded from check_stderr. Implementation specific flags are always set via kwargs.
If required to mask part of command from logging, log_mask_re attribute can be set global over instance or provided with command. All regex matched groups will be replaced by ‘<*masked*>’.
result: ExecResult = helper.execute(
command="AUTH='top_secret_key'; run command", # type: str
verbose=False, # type: bool
timeout=1 * 60 * 60, # type: typing.Optional[int]
log_mask_re=r"AUTH\s*=\s*'(\w+)'" # type: typing.Optional[str]
)result.cmd will be equal to AUTH=’<*masked*>’; run command
ExecResult
Execution result object has a set of useful properties:
cmd - Command
exit_code - Command return code. If possible to decode using enumerators for Linux -> it used.
stdin -> str. Text representation of stdin.
stdout -> typing.Tuple[bytes]. Raw stdout output.
stderr -> typing.Tuple[bytes]. Raw stderr output.
stdout_bin -> bytearray. Binary stdout output.
stderr_bin -> bytearray. Binary stderr output.
stdout_str -> str. Text representation of output.
stderr_str -> str. Text representation of output.
stdout_brief -> str. Up to 7 lines from stdout (3 first and 3 last if >7 lines).
stderr_brief -> str. Up to 7 lines from stderr (3 first and 3 last if >7 lines).
stdout_json - STDOUT decoded as JSON.
stdout_yaml - STDOUT decoded as YAML. Accessible only if PyYAML or ruamel.YAML library installed. (Extras: yaml)
stdout_xml - STDOUT decoded as XML to ElementTree using defusedxml library. Accessible only if defusedxml library installed. (Extras: xml)
stdout_lxml - STDOUT decoded as XML to ElementTree using lxml library. Accessible only if lxml library installed. (Extras: lxml) Can be insecure.
timestamp -> typing.Optional(datetime.datetime). Timestamp for received exit code.
SSHClient specific
SSHClient commands support get_pty flag, which enables PTY open on remote side. PTY width and height can be set via keyword arguments, dimensions in pixels are always 0x0.
Possible to call commands in parallel on multiple hosts if it’s not produce huge output:
results: Dict[Tuple[str, int], ExecResult] = SSHClient.execute_together(
remotes, # type: typing.Iterable[SSHClient]
command, # type: str
timeout=1 * 60 * 60, # type: type: typing.Union[int, float, None]
expected=(0,), # type: typing.Iterable[typing.Union[int, ExitCodes]]
raise_on_err=True, # type: bool
# Keyword only:
stdin=None, # type: typing.Union[bytes, str, bytearray, None]
open_stdout=True, # type: bool
open_stderr=True, # type: bool
log_mask_re=None, # type: typing.Optional[str]
exception_class=ParallelCallProcessError # typing.Type[ParallelCallProcessError]
)
results # type: typing.Dict[typing.Tuple[str, int], exec_result.ExecResult]Results is a dict with keys = (hostname, port) and and results in values. By default execute_together raises exception if unexpected return code on any remote.
To open new connection using current as proxy is accessible method proxy_to. Basic usage example:
conn: SSHClient = client.proxy_to(host, username="username", password="password")For execute through SSH host can be used execute_through_host method:
result: ExecResult = client.execute_through_host(
hostname, # type: str
command, # type: str
# Keyword only:
auth=None, # type: typing.Optional[SSHAuth]
target_port=22, # type: int
timeout=1 * 60 * 60, # type: type: typing.Union[int, float, None]
verbose=False, # type: bool
stdin=None, # type: typing.Union[bytes, str, bytearray, None]
open_stdout=True, # type: bool
open_stderr=True, # type: bool
log_mask_re=None, # type: typing.Optional[str]
get_pty=False, # type: bool
width=80, # type: int
height=24 # type: int
)Where hostname is a target hostname, auth is an alternate credentials for target host.
SSH client implements fast sudo support via context manager:
Commands will be run with sudo enforced independently from client settings for normal usage:
with client.sudo(enforce=True):
...Commands will be run without sudo independently from client settings for normal usage:
with client.sudo(enforce=False):
...“Permanent client setting”:
client.sudo_mode = mode # where mode is True or FalseSSH Client supports sFTP for working with remote files:
with client.open(path, mode='r') as f:
...For fast remote paths checks available methods:
exists(path) -> bool
>>> conn.exists('/etc/passwd')
Truestat(path) -> paramiko.sftp_attr.SFTPAttributes
>>> conn.stat('/etc/passwd')
<SFTPAttributes: [ size=1882 uid=0 gid=0 mode=0o100644 atime=1521618061 mtime=1449733241 ]>
>>> str(conn.stat('/etc/passwd'))
'-rw-r--r-- 1 0 0 1882 10 Dec 2015 ?'isfile(path) -> bool
>>> conn.isfile('/etc/passwd')
Trueisdir(path) -> bool
>>> conn.isdir('/etc/passwd')
FalseAdditional (non-standard) helpers:
mkdir(path: str) - execute mkdir -p path
rm_rf(path: str) - execute rm -rf path
upload(source: str, target: str) - upload file or from source to target using sFTP.
download(destination: str, target: str) - download file from target to destination using sFTP.
Subprocess specific
Keyword arguments:
cwd - working directory.
env - environment variables dict.
async_api.Subprocess specific
All standard methods are coroutines. Async context manager also available.
Example:
async with helper:
result: ExecResult = await helper.execute(
command, # type: str
verbose=False, # type: bool
timeout=1 * 60 * 60, # type: typing.Union[int, float, None]
**kwargs
)Testing
The main test mechanism for the package exec-helpers is using tox. Available environments can be collected via tox -l
CI systems
For code checking several CI systems is used in parallel:
Travis CI: is used for checking: PEP8, pylint, bandit, installation possibility and unit tests. Also it’s publishes coverage on coveralls.
Azure Pipelines: is used for windows compatibility checking.
coveralls: is used for coverage display.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file exec-helpers-6.0.0rc3.tar.gz.
File metadata
- Download URL: exec-helpers-6.0.0rc3.tar.gz
- Upload date:
- Size: 70.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a6a04edcf84df7a3df2522fab84074980e0ff0b25e0d9dd0e4dd77131d7ca440 |
|
| MD5 | fb7101ad036d01d1eb40b88d91739335 |
|
| BLAKE2b-256 | 661f4e07e215315207f8c03a9e675e2537a2e367bb6bbefea7bc0e56888368f5 |
File details
Details for the file exec_helpers-6.0.0rc3-py3-none-any.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-py3-none-any.whl
- Upload date:
- Size: 61.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3b4b6fb484760d61ae674bf6c9032f806fa4ed20801948daeb492ecbd60bb709 |
|
| MD5 | 8afa0a459ba5b39ba01d2a1d11b6480d |
|
| BLAKE2b-256 | 77810a5a37e928ab92570c3b95b04f585b7fabe78e582cb70aa702886b06dd48 |
File details
Details for the file exec_helpers-6.0.0rc3-cp38-cp38m-win_amd64.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp38-cp38m-win_amd64.whl
- Upload date:
- Size: 938.1 kB
- Tags: CPython 3.8m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.8.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a1008a6fb9ef2409885d190ff4bebaae6fa3bf161f74ae4337f6e99fae39bab0 |
|
| MD5 | ada509355fb4549d0dc319829b75bf02 |
|
| BLAKE2b-256 | 6873310ace17d89a66c2dd2cdefbd6481bb237c4f67976738870344e347e8fea |
File details
Details for the file exec_helpers-6.0.0rc3-cp38-cp38m-win32.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp38-cp38m-win32.whl
- Upload date:
- Size: 801.9 kB
- Tags: CPython 3.8m, Windows x86
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.8.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 22b931dec5f1d5469e85c39f8ffd740ae98c3480c5cbdc8f68337aa73672511c |
|
| MD5 | 9db80924eb9819cdbdb4c1582e1e57e1 |
|
| BLAKE2b-256 | 43c4501dc5bfd80ec57d1ee44f02a4cbb4c0f9b961910cd33796402f00820339 |
File details
Details for the file exec_helpers-6.0.0rc3-cp38-cp38-manylinux1_x86_64.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp38-cp38-manylinux1_x86_64.whl
- Upload date:
- Size: 4.2 MB
- Tags: CPython 3.8
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ad13efd20319842cfdd0d95ffeedcb8d88f9c6025757e9a53e78a04f732344fb |
|
| MD5 | 4c463a9299881a6c5c40d41990a8839c |
|
| BLAKE2b-256 | 732e10a519568739d4dec54d33f18f92b0659aa1db169c74fa67ab64cc158e26 |
File details
Details for the file exec_helpers-6.0.0rc3-cp37-cp37m-win_amd64.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp37-cp37m-win_amd64.whl
- Upload date:
- Size: 887.3 kB
- Tags: CPython 3.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | bf55a212a7188fb4551377a725c8a7fa55d555ad11ef0b4a879cc801e613216a |
|
| MD5 | a0cef804816a92f6b1925419212b1a57 |
|
| BLAKE2b-256 | 51b991bc9231e0b18b2ab3cc2e03f22f496dac74af3eb30cad35049608901bc4 |
File details
Details for the file exec_helpers-6.0.0rc3-cp37-cp37m-win32.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp37-cp37m-win32.whl
- Upload date:
- Size: 762.5 kB
- Tags: CPython 3.7m, Windows x86
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4ce0fcf45de196f38098d0832aea1740f151281420b88b64fba6b9d410ce51f0 |
|
| MD5 | 11818583ef06007912355c753bb7704f |
|
| BLAKE2b-256 | 676cd627566c6b9027e000697d816db09a5f2dfb9b1c7d1e1912e9d9df65ad79 |
File details
Details for the file exec_helpers-6.0.0rc3-cp37-cp37m-manylinux1_x86_64.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp37-cp37m-manylinux1_x86_64.whl
- Upload date:
- Size: 3.8 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3834a7ea1f93241b4f3809c5350e2f52894cf9c408ff7866bc71b58457664bd1 |
|
| MD5 | ee6728a7cc1c53d2ed6f40905fe51ddf |
|
| BLAKE2b-256 | 2117bc8eae651db9c5b71d12130e7225cb71bb96ef5bbb4e16aaa7fd868fec37 |
File details
Details for the file exec_helpers-6.0.0rc3-cp36-cp36m-win_amd64.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp36-cp36m-win_amd64.whl
- Upload date:
- Size: 887.3 kB
- Tags: CPython 3.6m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f6c5b27fadbf9d8c400423d691df48cc5ba860a8bb671c69e735ba6666804b5c |
|
| MD5 | 1e39e07d8140b79e13bdf15b04fc1649 |
|
| BLAKE2b-256 | 24a0cece26ad6943bab0b277fc0d68782427f9b792886acc5a7336689304830d |
File details
Details for the file exec_helpers-6.0.0rc3-cp36-cp36m-win32.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp36-cp36m-win32.whl
- Upload date:
- Size: 762.7 kB
- Tags: CPython 3.6m, Windows x86
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 102c2729d27247f8070549704f0510a165a9638ad754ba5ec86dc154ea824724 |
|
| MD5 | d6da7cbe4a2a15bfe393ac24d381ded7 |
|
| BLAKE2b-256 | 3c81a74669d23a0b607d85cf65c44dedde015b788d9cb352ae1dce815666f28c |
File details
Details for the file exec_helpers-6.0.0rc3-cp36-cp36m-manylinux1_x86_64.whl.
File metadata
- Download URL: exec_helpers-6.0.0rc3-cp36-cp36m-manylinux1_x86_64.whl
- Upload date:
- Size: 3.8 MB
- Tags: CPython 3.6m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.7.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cf0be3dcec0a3c13a2dc59bc89b1079373e15bcfb5db55f44b440e9e4030ca00 |
|
| MD5 | 728e6e79630cc8a5187a5ce24a041599 |
|
| BLAKE2b-256 | 11c56719fa6b2dd99b1f5295de6baff9e69b14544d25575dd338878ff6f0fdbc |
