tmt 1.3.1.1

Description

The tmt python module and command-line tool implement the Metadata Specification which allows storing all needed test execution data directly within a git repository. In this way, it makes testing independent on any external test management system.

The Flexible Metadata Format fmf is used to store data in both human and machine readable way close to the source code. Thanks to inheritance and elasticity metadata are organized in the structure efficiently, preventing unnecessary duplication.

The tool provides a user-friendly way to create, debug and easily run tests from your laptop across different environments. It also allows to easily convert old metadata, list and filter available tests and verify them against the L1 specification.

Plans are used to group tests and configure individual test steps defined by the L2 specification. They describe how to select tests for execution, how to provision the environment, how to prepare it for testing or how the test results should be reported.

Stories, defined by the L3 specification, can be used to track implementation, test and documentation coverage for individual features or requirements. Thanks to this you can track everything in one place, including the project implementation progress.

Synopsis

Command line usage is straightforward:

tmt command [options]

Examples

Let’s see which tests, plans and stories are available:

tmt

Initialize the metadata tree in the current directory, optionally with example content based on templates:

tmt init
tmt init --template base

Run all or selected steps for each plan:

tmt run
tmt run discover
tmt run prepare execute

List tests, show details, check against the specification:

tmt test ls
tmt test show
tmt test lint

Create a new test, import test metadata from other formats:

tmt test create
tmt test import

List plans, show details, check against the specification:

tmt plan ls
tmt plan show
tmt plan lint

List stories, check details, show coverage status:

tmt story ls
tmt story show
tmt story coverage

Many commands support regular expression filtering and other specific options:

tmt story ls cli
tmt story show create
tmt story coverage --implemented

Check help message of individual commands for the full list of available options.

Options

Here is the list of the most frequently used commands and options.

Install

The main tmt package provides the core features with a minimal set of dependencies:

sudo dnf install tmt

In order to enable additional functionality, such as particular provision or report plugins, install the respective subpackage:

sudo dnf install tmt-test-convert
sudo dnf install tmt-report-html
sudo dnf install tmt-provision-container
sudo dnf install tmt-provision-virtual

If you don’t care about disk space and want to have all available features right at hand install everything:

sudo dnf install tmt-all

For RHEL 8 and CentOS 8, first make sure that you have enabled the EPEL repository:

sudo dnf install epel-release
sudo dnf install tmt

Impatient to try the fresh features as soon as possible? Install the latest greatest version from the copr repository:

sudo dnf copr enable psss/tmt
sudo dnf install tmt

Not sure, just want to try out how it works? Experiment safely and easily inside a container:

podman run -it --rm quay.io/testing-farm/tmt bash
podman run -it --rm quay.io/testing-farm/tmt-all bash

When installing using pip you might need to install additional packages on your system:

sudo dnf install gcc {python3,libvirt,krb5,libpq}-devel
pip install --user tmt

Note: You can omit the --user flag if in a virtual environment.

Develop

In order to experiment, play with the latest bits and develop improvements it is best to use a virtual environment:

mkvirtualenv tmt
git clone https://github.com/psss/tmt
cd tmt
pip install -e .

Install python3-virtualenvwrapper to easily create and enable virtual environments using mkvirtualenv and workon. Note that if you have freshly installed the package you need to open a new shell session to enable the wrapper functions.

The main tmt package contains only the core dependencies. For building documentation, testing changes, importing/exporting test cases or advanced provisioning options install the extra deps:

pip install '.[docs]'
pip install '.[tests]'
pip install '.[convert]'
pip install '.[provision]'

Or simply install all extra dependencies to make sure you have everything needed for the tmt development ready on your system:

pip install '.[all]'

Exit Codes

The following exit codes are returned from tmt run. Note that you can use the --quiet option to completely disable output and only check for the exit code.

0

At least one test passed, there was no fail, warn or error.

1

There was a fail or warn identified, but no error.

2

Errors occured during test execution.

3

No test results found.

Variables

The following environment variables can be used to modify behaviour of the tmt command.

TMT_DEBUG

Enable the desired debug level. Most of the commands support levels from 1 to 3. However, some of the plugins go even deeper when needed.

Links

Git: https://github.com/psss/tmt

Docs: http://tmt.readthedocs.io/

Stories: https://tmt.readthedocs.io/en/latest/stories.html

Issues: https://github.com/psss/tmt/issues

Releases: https://github.com/psss/tmt/releases

Copr: http://copr.fedoraproject.org/coprs/psss/tmt

PIP: https://pypi-hypernode.com/project/tmt/

Travis: https://travis-ci.org/psss/tmt

Coveralls: https://coveralls.io/github/psss/tmt

Metadata Specification: https://tmt.readthedocs.io/en/latest/spec.html

Flexible Metadata Format: http://fmf.readthedocs.io/

Packit & Testing Farm: https://packit.dev/testing-farm/

Authors

Petr Šplíchal, Miro Hrončok, Alexander Sosedkin, Lukáš Zachar, Petr Menšík, Leoš Pol, Miroslav Vadkerti, Pavel Valena, Jakub Heger, Honza Horák, Rachel Sibley, František Nečas, Michal Ruprich, Martin Kyral, Miloš Prchlík, Tomáš Navrátil, František Lachman, Patrik Kis, Ondrej Mosnáček, Andrea Ficková, Denis Karpelevich and Michal Srb.

Copyright

Copyright (c) 2019 Red Hat, Inc.

This program is free software; you can redistribute it and/or modify it under the terms of the MIT License.