pytest-random-order 0.5.5

Quick Start

$ pip install pytest-random-order

The plugin is enabled by default. To randomise the order of tests within modules and shuffle the order of test modules (which is the default behaviour of the plugin), just run pytest as always:

$ pytest -v

To change the level of randomness allowed, run pytest with --random-order-bucket=<bucket-type> option where <bucket-type> can be class, module, package, or global:

$ pytest -v --random-order-bucket=package

To disable reordering of tests in a module or class, use pytest marker notation:

pytestmark = pytest.mark.random_order(disabled=True)

To rerun tests in a particular order:

$ pytest -v --random-order-seed=<value-reported-in-previous-run>

Design

pytest-random-order plugin groups tests in buckets, shuffles them within buckets and then shuffles the buckets.

You can choose from four types of buckets:

class

Tests will be shuffled within a class and classes will be shuffled, but tests from one class will never have tests from other classes or modules run in-between them.

module

Same as above at module level. This is the default setting.

package

Same as above at package level. Note that modules (and hence tests inside those modules) that belong to package x.y.z do not belong to package x.y, so they will fall in different buckets when randomising with package bucket type.

global

All tests fall in the same bucket, full randomness, tests probably take longer to run.

If you have three buckets of tests A, B, and C with three tests 1 and 2, and 3 in each of them, then one of many potential orderings that non-global randomisation can produce could be:

c2, c1, c3, a3, a1, a2, b3, b2, b1

As you can see, all C tests are executed “next” to each other and so are tests in buckets A and B. Tests from any bucket X are guaranteed to not be interspersed with tests from another bucket Y. For example, if you choose bucket type module then bucket X contains all tests that are in this module.

By default, your tests will be randomised at module level which means that tests within a single module X will be executed in no particular order, but tests from other modules will not be mixed in between tests of module X.

The randomised reordering can be disabled per module or per class irrespective of the chosen bucket type.

Usage and Tips

pytestmark = pytest.mark.random_order(disabled=True)

def test_number_one():
    assert True

def test_number_two():
    assert True

class MyTest(TestCase):
    pytestmark = pytest.mark.random_order(disabled=True)

    def test_number_one(self):
        self.assertTrue(True)

============================= test session starts ==============================
..
Using --random-order-bucket=module
Using --random-order-seed=24775
...

$ pytest -v --random-order-seed=24775

$ pytest -p no:random-order -v

License

Distributed under the terms of the MIT license, “pytest-random-order” is free and open source software