Skip to main content

testrig: running tests for dependent packages

Project description

https://travis-ci.org/pv/testrig.png?branch=master

testrig

Python package integration testing.

Runs the test suite of some package against ‘old’ and ‘new’ versions of given dependencies. Failures that appear in ‘new’ are reported.

Each test suite run is run in a virtualenv constructed from scratch. You should install ccache (and possibly also f90cache) to avoid drinking too much coffee.

Alternatively, binary conda packages can be used — however, binary incompatibities may arise in this configuration.

Currently, this is POSIX-only, and tested only on Linux.

Usage

Run:

python -mtestrig --help
python -mtestrig examples/testrig.ini pandas       # run tests
python -mtestrig examples/testrig-conda.ini pandas # use conda packages
python -mtestrig examples/testrig.ini -j           # run all packages parallel

The runs may take a long time, as it builds everything from source.

Configuration

Configuration is read from an .ini file. It contains sections, one per test environment. Section named DEFAULT can be used to specify (overridable) default values for the configuration items.

An example first (runs scipy test suite against old and new numpy versions):

[DEFAULT]
pkgs = nose tempita Cython==0.22 scipy==0.17.0
old = numpy==1.7.2 {pkgs}
new = Cython==0.22 git+https://github.com/numpy/numpy@master {pkgs}

[scipy]
pkgs = {pkgs} scipy
run = python -mpytest --junit-xml=junit.xml --pyarg scipy
parser = junit:junit.xml
envvars =
    SETUPCFG=$DIR/mysetup.cfg

More examples can be found under the examples/ folder.

The configuration items in each section are:

  • env: which environment to use

    • virtualenv: virtualenv + pip, all packages are built from sources

    • conda: conda, uses binary packages, except for git+ urls and package names prefixed by pip+. Note that you may need to write stuff like numpy git+https://github.com/numpy/numpy.git since conda only understand that packages installed by it are present.

  • old: package specifications for the ‘old’ configuration (see below).

  • new: package specifications for the ‘new’ configuration (see below).

  • run: command that runs the tests.

  • parser: parser for the test output. Available options:

    • junit:FILENAME: parses xUnit/jUnit xml result output, stored in given filename. Both nose and py.test can produce this output: py.test --junit-xml=junit.xml ... and nosetests --with-xunit --xunit-file=junit.xml ....

    • nose: parses nose stdout

  • envvars: additional environment variables to set (also for pip install). The text $DIR is replaced by an absolute path of the directory where the configuration file resides.

The values support string interpolation, and default values can be specified in the DEFAULT section. For example:

[DEFAULT]
pre = foo
post = quux

[section]
post = {post} quux2
new = {pre} bar {post}

produces the value new = foo bar quux quux2.

The package specifications are a string containing a list of pip (or conda if using env=conda) packages version specifications, with the following additional possible items:

  • --no-binary: do not install following packages via wheels or conda.

  • --binary: install following packages via wheels or conda, if possible.

By default, binary packages are used.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

testrig-0.2.tar.gz (15.2 kB view details)

Uploaded Source

File details

Details for the file testrig-0.2.tar.gz.

File metadata

  • Download URL: testrig-0.2.tar.gz
  • Upload date:
  • Size: 15.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for testrig-0.2.tar.gz
Algorithm Hash digest
SHA256 fe6088d9fcac8bddcf16e3749897d54aab548a814777c1e410cef0c9530c54b1
MD5 5fb1111b93758a0567b6014fc3668452
BLAKE2b-256 a442f8b0624c46c44b88d1652d27a0b9fbfe5c01a5c1872d23e6057cbb021a94

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page