tmt 1.28.0

Description

The tmt tool provides a user-friendly way to work with tests. You can comfortably create new tests, safely and easily run tests across different environments, review test results, debug test code and enable tests in the CI using a consistent and concise config.

The python module and command-line tool implement the Metadata Specification which allows storing all needed test execution data directly within a git repository. Together with possibility to reference remote repositories it makes it easy to share test coverage across projects and distros.

The Flexible Metadata Format fmf is used to store data in both human and machine readable way close to the source code. Thanks to inheritance and elasticity metadata are organized in the structure efficiently, preventing unnecessary duplication.

Specification

There are several metadata levels defined by the specification:

Core attributes such as summary or description which are common across all levels are defined by the special L0 metadata.

Tests, or L1 metadata, define attributes which are closely related to individual test cases such as test script, framework, directory path where the test should be executed, maximum test duration or packages required to run the test.

Plans, also called L2 metadata, are used to group relevant tests and enable them in the CI. They describe how to discover tests for execution, how to provision the environment, how to prepare it for testing, how to execute tests and report test results.

Stories, which implement the L3 metadata, can be used to track implementation, test and documentation coverage for individual features or requirements. Thanks to this you can track everything in one place, including the project implementation progress.

Synopsis

Command line usage is straightforward:

tmt command [options]

Examples

Let’s see which tests, plans and stories are available:

tmt

Initialize the metadata tree in the current directory, optionally with example content based on templates:

tmt init
tmt init --template base

Run all or selected steps for each plan:

tmt run
tmt run discover
tmt run prepare execute

List tests, show details, check against the specification:

tmt tests ls
tmt tests show
tmt tests lint

Create a new test, import test metadata from other formats:

tmt test create
tmt test import

List plans, show details, check against the specification:

tmt plans ls
tmt plans show
tmt plans lint

List stories, check details, show coverage status:

tmt stories ls
tmt stories show
tmt stories coverage

Many commands support regular expression filtering and other specific options:

tmt stories ls cli
tmt stories show create
tmt stories coverage --implemented

Check help message of individual commands for the full list of available options.

Options

Here is the list of the most frequently used commands and options.

Install

The main tmt package provides the core features with a minimal set of dependencies:

sudo dnf install tmt

In order to enable additional functionality, such as particular provision or report plugins, install the respective subpackage:

sudo dnf install tmt+test-convert
sudo dnf install tmt+provision-container
sudo dnf install tmt+provision-virtual

If you don’t care about disk space and want to have all available features right at hand install everything:

sudo dnf install tmt+all

For CentOS and RHEL, first make sure that you have available the EPEL repository. You might also have to enable additional repositories:

sudo dnf config-manager --enable crb         # CentOS 9
sudo dnf config-manager --enable rhel-CRB    # RHEL 9
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

sudo dnf install tmt

For plugins which cannot work outside of VPN and so live within its walls you need to enable the internal copr repository first. Then you can install either everything or only those you need:

sudo dnf install tmt-redhat-all
sudo dnf install tmt-redhat-*

Impatient to try the fresh features as soon as possible? Install the latest greatest version from the copr repository:

sudo dnf copr enable @teemtee/tmt
sudo dnf install tmt

Not sure, just want to try out how it works? Experiment safely and easily inside a container:

podman run -it --rm quay.io/testing-farm/tmt bash
podman run -it --rm quay.io/testing-farm/tmt-all bash

When installing using pip you might need to install additional packages on your system:

sudo dnf install gcc redhat-rpm-config
sudo dnf install {python3,libvirt,krb5,libpq}-devel
pip install --user tmt

On other distributions than Fedora or RHEL the package names might be different. For example on Ubuntu to install all packages to have provision plugins working:

sudo apt install libkrb5-dev pkg-config libvirt-dev genisoimage qemu-kvm libvirt-daemon-system
pip install --user "tmt[provision]"

Note: You can omit the --user flag if in a virtual environment.

Shell Completion

The rpm package includes a system wide script which enables the command line completion for bash so no additional config should be needed. If you use a different installation method or prefer another shell, see the instructions below.

For Bash, add this to ~/.bashrc:

eval "$(_TMT_COMPLETE=source_bash tmt)"

For Zsh, add this to ~/.zshrc:

eval "$(_TMT_COMPLETE=source_zsh tmt)"

For Fish, add this to ~/.config/fish/completions/tmt.fish:

eval (env _TMT_COMPLETE=source_fish tmt)

Open a new shell to enable completion. Or run the eval command directly in your current shell to enable it temporarily.

This is however run every time you start a shell which can cause some delay. To speed it up, write the generated script to a file and then source it from your shell’s configuration file. All of this can be achieved using tmt setup completion command. By default, it outputs the completion script to the terminal but it can also add it to your ~/.bashrc or ~/.zshrc using the --install option:

tmt setup completion {bash, zsh, fish} --install

Exit Codes

The following exit codes are returned from tmt run. Note that you can use the --quiet option to completely disable output and only check for the exit code.

0

At least one test passed, there was no fail, warn or error.

1

There was a fail or warn identified, but no error.

2

Errors occured during test execution.

3

No test results found.

Variables

The list of available environment variables which can be used to adjust the execution.

TMT_PLUGIN_EXECUTE_TMT_VERBOSE=3 tmt run ... execute -h tmt ...

TMT_PLUGIN_EXECUTE_TMT_VERBOSE=3 tmt run ... execute -h tmt -v ...

discover:
    how: fmf
    url: https://example.org/

tmt run ... discover -h fmf --url https://actual.org/ ...

TMT_PLUGIN_DISCOVER_FMF_URL=https://actual.org/ tmt run ...

TMT_PLUGIN_EXECUTE_TMT_INTERACTIVE=1 tmt run ... execute -h tmt ...

Links

Git: https://github.com/teemtee/tmt

Docs: http://tmt.readthedocs.io/

Stories: https://tmt.readthedocs.io/en/stable/stories.html

Issues: https://github.com/teemtee/tmt/issues

Releases: https://github.com/teemtee/tmt/releases

Copr: https://copr.fedorainfracloud.org/coprs/g/teemtee/tmt/

PIP: https://pypi-hypernode.com/project/tmt/

Metadata Specification: https://tmt.readthedocs.io/en/stable/spec.html

Flexible Metadata Format: http://fmf.readthedocs.io/

Packit & Testing Farm: https://packit.dev/testing-farm/

Authors

Petr Šplíchal, Miro Hrončok, Alexander Sosedkin, Lukáš Zachar, Petr Menšík, Leoš Pol, Miroslav Vadkerti, Pavel Valena, Jakub Heger, Honza Horák, Rachel Sibley, František Nečas, Michal Ruprich, Martin Kyral, Miloš Prchlík, Tomáš Navrátil, František Lachman, Patrik Kis, Ondrej Mosnáček, Andrea Ficková, Denis Karpelevich, Michal Srb, Jan Ščotka, Artem Zhukov, Vinzenz Feenstra, Inessa Vasilevskaya, Štěpán Němec, Robin Hack, Yulia Kopkova, Ondrej Moriš, Martin Zelený, Karel Šrot, František Zatloukal, Simon Walter, Petr Matyáš, Yariv Rachmani, Pavel Cahyna, Martin Litwora, Brian Grech, Vojtěch Eichler, Philip Daly, Vector Li, Evgeny Fedin, Guy Inger, Adrián Tomašov, Jan Havlín, Lukáš Kotek, Daniel Diblík, Laura Barcziova, Marián Konček, Marcin Sobczyk, Jiří Jabůrek, Huijing Hei, Tibor Dudlák, Jan Macků, Filip Vágner, Martin Hoyer, Iveta Česalová, Yi Zhang, Zhaojuan Guo, Natália Bubáková, Michal Josef Špaček, Jiří Popelka, Matej Focko, Yulia Kopkova and Tomáš Bajer.

Copyright

Copyright Red Hat

This program is free software; you can redistribute it and/or modify it under the terms of the MIT License.