Skip to main content

Execution helpers for simplified usage of subprocess and ssh.

Project description

exec-helpers

https://travis-ci.org/python-useful-helpers/exec-helpers.svg?branch=master https://coveralls.io/repos/github/python-useful-helpers/exec-helpers/badge.svg?branch=master Documentation Status https://img.shields.io/pypi/v/exec-helpers.svg https://img.shields.io/pypi/pyversions/exec-helpers.svg https://img.shields.io/pypi/status/exec-helpers.svg https://img.shields.io/github/license/python-useful-helpers/exec-helpers.svg

Execution helpers for simplified usage of subprocess and ssh. Why another subprocess wrapper and why no clear paramiko?

Historically paramiko offers good ssh client, but with specific limitations: you can call command with timeout, but without receiving return code, or call command and wait for return code, but without timeout processing.

In the most cases, we are need just simple SSH client with comfortable API for calls, calls via SSH proxy and checking return code/stderr. This library offers this functionality with connection memorizing, deadlock free polling and friendly result objects (with inline decoding of YAML, JSON, binary or just strings). In addition this library offers the same API for subprocess calls, but with specific limitation: no parallel calls (for protection from race conditions).

Pros:

Python 2.7
Python 3.4
Python 3.5
Python 3.6
PyPy
PyPy3 3.5+

This package includes:

  • SSHClient - historically the first one helper, which used for SSH connections and requires memorization due to impossibility of connection close prediction. Several API calls for sFTP also presents.

  • SSHAuth - class for credentials storage. SSHClient does not store credentials as-is, but uses SSHAuth for it. Objects of this class can be copied between ssh connection objects, also it used for execute_through_host.

  • Subprocess - subprocess.Popen wrapper with timeouts, polling and almost the same API, as SSHClient (except specific flags, like cwd for subprocess and get_tty for ssh).

  • ExecResult - class for execution results storage. Contains exit code, stdout, stderr and getters for decoding as JSON, YAML, string, bytearray and brief strings (up to 7 lines).

  • ExitCodes - enumerator for standard Linux exit codes. BASH return codes (broduced from signal codes) also available.

Usage

SSHClient

Basic initialization of SSHClient can be done without construction of specific objects:

client = exec_helpers.SSHClient(host, username="username", password="password")

If ssh agent is running - keys will be collected by paramiko automatically, but if keys are in specific location - it should be loaded manually and provided as iterable object of paramiko.RSAKey.

For advanced cases or re-use of credentials, SSHAuth object should be used. It can be collected from connection object via property auth.

Creation from scratch:

auth = exec_helpers.SSHAuth(
    username='username',  # type: typing.Optional[str]
    password='password',  # type: typing.Optional[str]
    key=None,  # type: typing.Optional[paramiko.RSAKey]
    keys=None,  # type: typing.Optional[typing.Iterable[paramiko.RSAKey]],
    key_filename=None,  # type: typing.Union[typing.List[str], str, None]
    passphrase=None,  # type: typing.Optional[str]
)

Key is a main connection key (always tried first) and keys are alternate keys. Key filename is afilename or list of filenames with keys, which should be loaded. Passphrase is an alternate password for keys, if it differs from main password. If main key now correct for username - alternate keys tried, if correct key found - it became main. If no working key - password is used and None is set as main key.

Context manager is available, connection is closed and lock is released on exit from context.

Subprocess

No initialization required. Context manager is available, subprocess is killed and lock is released on exit from context.

Base methods

Main methods are execute, check_call and check_stderr for simple executing, executing and checking return code and executing, checking return code and checking for empty stderr output. This methods are almost the same for SSHCleint and Subprocess, except specific flags.

result = helper.execute(
    command,  # type: str
    verbose=False,  # type: bool
    timeout=1 * 60 * 60,  # type: typing.Optional[int]
    **kwargs
)
result = helper.check_call(
    command,  # type: str
    verbose=False,  # type: bool
    timeout=1 * 60 * 60,  # type: typing.Optional[int]
    error_info=None,  # type: typing.Optional[str]
    expected=None,  # type: typing.Optional[typing.Iterable[int]]
    raise_on_err=True,  # type: bool
    **kwargs
)
result = helper.check_stderr(
    command,  # type: str
    verbose=False,  # type: bool
    timeout=1 * 60 * 60,  # type: typing.Optional[int]
    error_info=None,  # type: typing.Optional[str]
    raise_on_err=True,  # type: bool
)

If no STDOUT or STDERR required, it is possible to disable this FIFO pipes via **kwargs with flags open_stdout=False and open_stderr=False.

The next command level uses lower level and kwargs are forwarded, so expected exit codes are forwarded from check_stderr. Implementation specific flags are always set via kwargs.

If required to mask part of command from logging, log_mask_re attribute can be set global over instance or providden with command. All regex matched groups will be replaced by ‘<*masked*>’.

result = helper.execute(
    command="AUTH='top_secret_key'; run command",  # type: str
    verbose=False,  # type: bool
    timeout=1 * 60 * 60,  # type: typing.Optional[int]
    log_mask_re=r"AUTH\s*=\s*'(\w+)'"  # type: typing.Optional[str]
)

result.cmd will be equal to AUTH=’<*masked*>’; run command

ExecResult

Execution result object has a set of useful properties:

  • cmd - Command

  • exit_code - Command return code. If possible to decode using enumerators for Linux -> it used.

  • stdin -> str. Text representation of stdin.

  • stdout -> typing.Tuple[bytes]. Raw stdout output.

  • stderr -> typing.Tuple[bytes]. Raw stderr output.

  • stdout_bin -> bytearray. Binary stdout output.

  • stderr_bin -> bytearray. Binary stderr output.

  • stdout_str -> six.text_types. Text representation of output.

  • stderr_str -> six.text_types. Text representation of output.

  • stdout_brief -> six.text_types. Up to 7 lines from stdout (3 first and 3 last if >7 lines).

  • stderr_brief -> six.text_types. Up to 7 lines from stderr (3 first and 3 last if >7 lines).

  • stdout_json - STDOUT decoded as JSON.

  • stdout_yaml - STDOUT decoded as YAML.

  • timestamp -> typing.Optional(datetime.datetime). Timestamp for received exit code.

SSHClient specific

SSHClient commands support get_pty flag, which enables PTY open on remote side. PTY width and height can be set via kwargs, dimensions in pixels are always 0x0.

Possible to call commands in parallel on multiple hosts if it’s not produce huge output:

results = SSHClient.execute_together(
    remotes,  # type: typing.Iterable[SSHClient]
    command,  # type: str
    timeout=1 * 60 * 60,  # type: typing.Optional[int]
    expected=None,  # type: typing.Optional[typing.Iterable[int]]
    raise_on_err=True  # type: bool
)
results  # type: typing.Dict[typing.Tuple[str, int], exec_result.ExecResult]

Results is a dict with keys = (hostname, port) and and results in values. By default execute_together raises exception if unexpected return code on any remote.

For execute through SSH host can be used execute_through_host method:

result = client.execute_through_host(
    hostname,  # type: str
    command,  # type: str
    auth=None,  # type: typing.Optional[SSHAuth]
    target_port=22,  # type: int
    timeout=1 * 60 * 60,  # type: typing.Optional[int]
    verbose=False,  # type: bool
    get_pty=False,  # type: bool
)

Where hostname is a target hostname, auth is an alternate credentials for target host.

SSH client implements fast sudo support via context manager: Commands will be run with sudo enforced independently from client settings for normal usage:

with client.sudo(enforce=True):
    ...

Commands will be run without sudo independently from client settings for normal usage:

with client.sudo(enforce=False):
    ...

“Permanent client setting”:

client.sudo_mode = mode  # where mode is True or False

SSH Client supports sFTP for working with remote files:

with client.open(path, mode='r') as f:
    ...

For fast remote paths checks available methods:

  • exists(path) -> bool

>>> conn.exists('/etc/passwd')
True
  • stat(path) -> paramiko.sftp_attr.SFTPAttributes

>>> conn.stat('/etc/passwd')
<SFTPAttributes: [ size=1882 uid=0 gid=0 mode=0o100644 atime=1521618061 mtime=1449733241 ]>
>>> str(conn.stat('/etc/passwd'))
'-rw-r--r--   1 0        0            1882 10 Dec 2015  ?'
  • isfile(path) -> bool

>>> conn.isfile('/etc/passwd')
True
  • isdir(path) -> bool

>>> conn.isdir('/etc/passwd')
False

Additional (non-standard) helpers:

  • mkdir(path: str) - execute mkdir -p path

  • rm_rf(path: str) - execute rm -rf path

  • upload(source: str, target: str) - upload file or from source to target using sFTP.

  • download(destination: str, target: str) - download file from target to destination using sFTP.

Subprocess specific

Kwargs set properties:

  • cwd - working directory.

  • env - environment variables dict.

Testing

The main test mechanism for the package exec-helpers is using tox. Available environments can be collected via tox -l

CI systems

For code checking several CI systems is used in parallel:

  1. Travis CI: is used for checking: PEP8, pylint, bandit, installation possibility and unit tests. Also it’s publishes coverage on coveralls.

  2. coveralls: is used for coverage display.

Project details


Release history Release notifications | RSS feed

This version

1.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

exec-helpers-1.2.0.tar.gz (600.7 kB view details)

Uploaded Source

Built Distributions

exec_helpers-1.2.0-cp36-cp36m-manylinux1_x86_64.whl (1.9 MB view details)

Uploaded CPython 3.6m

exec_helpers-1.2.0-cp36-cp36m-manylinux1_i686.whl (1.8 MB view details)

Uploaded CPython 3.6m

exec_helpers-1.2.0-cp35-cp35m-manylinux1_x86_64.whl (1.8 MB view details)

Uploaded CPython 3.5m

exec_helpers-1.2.0-cp35-cp35m-manylinux1_i686.whl (1.7 MB view details)

Uploaded CPython 3.5m

exec_helpers-1.2.0-cp34-cp34m-manylinux1_x86_64.whl (1.9 MB view details)

Uploaded CPython 3.4m

exec_helpers-1.2.0-cp34-cp34m-manylinux1_i686.whl (1.8 MB view details)

Uploaded CPython 3.4m

File details

Details for the file exec-helpers-1.2.0.tar.gz.

File metadata

File hashes

Hashes for exec-helpers-1.2.0.tar.gz
Algorithm Hash digest
SHA256 630b789ac0d368d849f8a1efe5a219b51b815d87397aafe952e9785f6bc92d5a
MD5 4d4f40c881bed13429b972e6ea739327
BLAKE2b-256 a2b02d3bee1b1feb7905d614fdb68016675d858d9be13262ca0d23e0962df99f

See more details on using hashes here.

File details

Details for the file exec_helpers-1.2.0-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for exec_helpers-1.2.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4fba1b73535e770a1c0a3f48759043ea8aaf41bd5d058b282b5f1b9dedde85ab
MD5 46287120db0014d85f9c73a618757558
BLAKE2b-256 d937ab4c081f2f748475d24884157ee7181004b57dc358bfaac8010881bba5c2

See more details on using hashes here.

File details

Details for the file exec_helpers-1.2.0-cp36-cp36m-manylinux1_i686.whl.

File metadata

File hashes

Hashes for exec_helpers-1.2.0-cp36-cp36m-manylinux1_i686.whl
Algorithm Hash digest
SHA256 870d71e17029349d4e057a6805fd56af5bc5cf272a3950ad6bbdec4657adbf3b
MD5 13c018523ade339d399f95c1cdef7e1d
BLAKE2b-256 734e0643d8d1bdafee1f54111f3b983753044c3b03bcd4fccbbf02414a49f17c

See more details on using hashes here.

File details

Details for the file exec_helpers-1.2.0-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for exec_helpers-1.2.0-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8e08315466c7f54e8558e2ead7c51b1c6bbb2f304fb34185300c4de3b0280af4
MD5 ceb909388400467e756b49042fbf3a81
BLAKE2b-256 00c35c11bc7ce342035445894cc85b06397e6afd94a2b50a4b53038f5d8e9aaf

See more details on using hashes here.

File details

Details for the file exec_helpers-1.2.0-cp35-cp35m-manylinux1_i686.whl.

File metadata

File hashes

Hashes for exec_helpers-1.2.0-cp35-cp35m-manylinux1_i686.whl
Algorithm Hash digest
SHA256 b239f83a184e1100eea6269ef532013ba2b411c8ef80470202e1bc8091d8b9ae
MD5 377f8256f831ab47ee0a7c1d7f68d629
BLAKE2b-256 5e524495bb934ad845fcf10957211a4d3af7786604daa8001cd51f414b63e2df

See more details on using hashes here.

File details

Details for the file exec_helpers-1.2.0-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for exec_helpers-1.2.0-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 57676f8d2f42fdbfd1a8045d0acca2ac6ff91dc125cce9554441bcf05975ecb7
MD5 d1ed689efe0f7ce305408cad99898273
BLAKE2b-256 314fd3ba21e1bc12f441feeb96b5a5a6b71ed4e145aa8eb9f676480fef043438

See more details on using hashes here.

File details

Details for the file exec_helpers-1.2.0-cp34-cp34m-manylinux1_i686.whl.

File metadata

File hashes

Hashes for exec_helpers-1.2.0-cp34-cp34m-manylinux1_i686.whl
Algorithm Hash digest
SHA256 54cdef1f91986d151fb2d8c9f573432e3f3e47bb3b402b03bc0e931b0d21905d
MD5 9e571d2ca60f4dca32a2a1ebdac824a8
BLAKE2b-256 7fd83dc4ef93819b89f65ebf8005d761129ff8d9c555076bbfbcc042d8fe1d2b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page