Toy project to test executables defined in a yaml file
Project description
A wrapper around py.test to run your things selectively.
Installation and running
Main requirements
UNIX like
Python 3.6, 3.7 or 3.8
From PyPA (pip install)
Install
The most recent packaged version can be installed with pip with options relevant to your usecase.
pip install test_exe_matrix
Run
test_exe_matrix [arguments...]
From source with poetry
Install
Note on poetry: you cannot install it from pip, you need to follow the instructions <https://python-poetry.org/docs/> _ on the official Web site.
git clone https://framagit.org/feth/test_exe_matrix.git
cd test_exe_matrix
poetry install
Run
poetry run test_exe_matrix [arguments...]
Command line options
Typical use:
[poetry run] test_exe_matrix
Synopsys:
usage: test_exe_matrix [-h] [-V] [--collect-only] [-v] [--markers]
[-m PYTEST_ARGS]
[testsuite [testsuite ...]]
Test command lines listed in yaml files.
positional arguments:
testsuite testsuite yaml file -see example
test_exe_matrix/matrix.yaml.
Argument can be specified multiple times
optional arguments:
-h, --help show this help message and exit
-V, --version show program's version number and exit
--collect-only Only collect tests, without running them.
-v Increase verbosity.
--markers Lists registered test markers.
-m PYTEST_ARGS Only run tests that match selection expression, ie. '-m "not
internet"'
test_exe_matrix is mainly a Py.test wrapper
-v adds very nice colors, thanks to py.test.
Parametrizing tests
Put your test suites in a yaml, like matrix.yaml (provided), or in several. Every yaml file will have a separate global config (markers declaration and general timeout).
Minimum viable test
- exe: /path/to/exe
The test will succeed if default values are met: see below (optional values).
Optional values
- timeout
Allowed running time in seconds. Program will be killed after this delay. Default to config/timeout or 1.
- retcode
Expected return code.
- stdout
Expected standard output (full match)
- stderr
Expected standard error (full match)
- partstdout
Substring expected in standard output.
- partstderr
Substring expected in standard error.
- args
List of arguments to your program.
- name
Description of the test being made.
- expect_too_long
Boolean: whether the program is expected to run over the time limit.
- markers
List of markers for the test. You can use standard py.test markers (especially ‘xfail’: indicating ‘expected failure’) or custom markers, that _must_ be declared in the config section.
Example
A complete (hopefully) example file is provided as “matrix.yaml”.
Given the examples, you may run
test_exe_matrix matrix.yaml --collect-only
to list the tests, and
test_exe_matrix matrix.yaml -v -m "not internet and not slow"
to deselect all tests requiring Internet and slow tests.
Dev: Build the package
The project uses poetry. Do
poetry build
And a pip installable wheel package will appear under dist/. For instance, you can install with:
pip3 install dist/test_exe_matrix-0.0.18-py3-none-any.whl
Licence
CeCILL 2.1 and GPL 2.1, see files LICENSE_CeCILL_2.1.txt and LICENSE_GPL_2.1.txt.
Thank you
Stéphane Bortzmeyer, who once asked if a simple tool testing the result of commands existed, and then reported bugs/wrote helpful tickets on framagit/gitlab to guide the development.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for test_exe_matrix-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5fce5a562c50aa5e737a27d52e028e832b67b192e0949d05d813d0ae34cfaa56 |
|
MD5 | f34506b2a3bda72141ea3e515aac742a |
|
BLAKE2b-256 | ec7ee34d925a47d27176deaea83d844ca0182869a3f47869477932220f2ac81d |