robotsuite 2.1.0

Setting robot variables from environment variables

Robot Framework supports overriding test variables from command-line, which is not-available when running tests as robotsuite-wrapped with other test runners. That’s why robotsuite supports settings variables as environment variables so that every ROBOT_-prefixed environment variable will be mapped into corresponding test variable without the ROBOT_-prefix.

Declaring tests non-critical by given set of tags

Robot Framework supports declaring tests with given tags as non-critical to prevent their failing to fail the complete build on CI. This is supported as keyword argument for RobotTestSuite as follows:

def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        layered(RobotTestSuite('mysuite.txt',
                               noncritical=['non-critical-tag']),
                layer=ACCEPTANCE_TESTING),
    ])
    return suite

Setting zope.testrunner-level

zope.testrunner supports annotating test suites with levels to avoid slow test being run unless wanted:

def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        layered(RobotTestSuite('mysuite.txt'),
                layer=ACCEPTANCE_TESTING),
    ])
    suite.level = 10
    return suite

Retry failing tests

You can retry a failed test. This can be useful for flaky robot browser tests. Warning: this may not be good for all types of test. For example any changes that were done in the test until the first failure, may persist.

You can enable retries in two ways:

• Set an environment variable ROBOTSUITE_RETRY_COUNT=X.

• Override this by passing retry_count=X to a RobotTestSuite call.

The default is zero: no retries. The retry count excludes the original try.

def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        robotsuite.RobotTestSuite('test_example.robot', retry_count=3),
        robotsuite.RobotTestSuite('test_variables.robot'),
        robotsuite.RobotTestSuite('test_setups', retry_count=2)
    ])
    return suite

Appending test results to existing test report

When running Robot Framework through robotsuite, its test reports are created into the current working directory with filenames robot_output.xml, robot_log.html and robot_report.html. The default behavior is to override the existing robot_output.xml (and also the other report files generated from that).

To merge test results from separate test runs into the same test report, set environment variable ROBOTSUITE_APPEND_OUTPUT_XML=1 to prevent robotsuite from overriding the existing test results, but to always append to the existing robot_output.xml.

Filtering test execution errors

Set environment variable ROBOTSUITE_LOGLEVEL=ERROR to filter all top level Test Execution Errors below the given log level (e.g. ERROR) from the merged test report. This is useful when unnecessary warnings are leaking from the tested code into Robot Framework logs.

Including or skipping all RobotTestSuite-wrapped tests

Robot Framework is often used with Selenium2Library to write acceptance test using the Selenium-framework. Yet, because those test may be slow to run, one might want sometimes (e.g. on CI) to run everything except the robotsuite wrapped tests, and later only the robotsuite wrapped tests.

This can be achieved for sure, with injecting a custom string into the names of robotsuite-wrapped tests with ROBOTSUITE_PREFIX-environment variable and then filter the test with that string.

E.g. run everything except the robotsuite wrapped tests with:

$ ROBOTSUITE_PREFIX=ROBOTSUITE bin/test --all -t \!ROBOTSUITE

and the other way around with:

$ ROBOTSUITE_PREFIX=ROBOTSUITE bin/test --all -t ROBOTSUITE

Re-using test suites from other packages

Sometime it could be useful to re-use acceptance test from some upstream package to test your slightly tailored package (e.g. with a custom theme). This can be done with by defining the test lookup location with package-keyword argment for RobotTestSuite:

def test_suite():
    suite = unittest.TestSuite()
    suite.addTests([
        layered(leveled(
            robotsuite.RobotTestSuite('robot',
                                      package='Products.CMFPlone.tests'),
        ), layer=PLONE_APP_MOSAIC_NO_PAC_ROBOT),
    ])
    return suite

2.1.0 (2020-02-13)

• Add optional support for retrying a failed test. Set environment variable ROBOTSUITE_RETRY_COUNT=X. Or override this by passing retry_count=X to a RobotTestSuite call. Default is zero: no retries. The retry count excludes the original try. [maurits]

• Add support for Robot Framework 3.2b1 [datakurre]

• Drop robotframework-python3 requirement, robotframework itself is already python 3 compatible. [gforcada]

2.0.0 (2016-12-22)

• Do not support python 2.6 anymore [gforcada]

• Remove unittest2 references [gforcada]

1.7.0 (2015-07-23)

• Add support for Robot Framework 2.9 [datakurre]

1.6.1 (2014-10-01)

• Add ROBOTSUITE_LOGLEVEL environment variable for filtering text execution errors during log merge [datakurre]

1.6.0 (2014-06-29)

• Add support for ROBOTSUITE_PREFIX-env for injecting custom string into all robotsuite test names [datakurre]

• Add UTF-8 support for robot variables [#6] [Tom Gross] [Vincent Fretin]

• Added Python 3.4 support [Vincent Fretin]

1.5.0 (2014-04-13)

• Restore original behavior to override the exiting robot_output.xml. Set environment variable ROBOTSUITE_APPEND_OUTPUT_XML=1 to append test results to the existing robot_output.xml. [datakurre]

1.4.3 (2014-01-27)

• Fix issue where test suites with sub-suites were not supported [datakurre]

1.4.2 (2013-11-22)

• Fix issue where suite setups were run for suites without tests wit robotframework >= 2.8.0 [datakurre]

1.4.1 (2013-10-13)

• Fix regression in 1.4.0, which dropped relative path names from robot test cases [datakurre]

• Fix RobotTestCase to mimic DocTestCase enough to allow plone.testing to set ‘layer’ for it [datakurre]

• Fix to set given test ‘setUp’ and ‘tearDown’ methods properly as instancemethods (will break existing methods not accepting ‘self’ parameters) [datakurre]

1.4.0 (2013-06-19)

• Supports only roboframework >= 2.8. Use robotsuite 1.3.x with robotframework 1.7.x [datakurre]

1.3.3 (2013-06-05)

• Fix to support custom screenshot filenames with paths [fix #2] [datakurre]

1.3.2 (2013-06-03)

• Remove screenshot capturing from functional test [datakurre]

1.3.1 (2013-06-03)

• Pin robotframework < 2.8rc1, because robotsuite is not yet compatible with robotframework 2.8.x [datakurre]

1.3.0 (2013-04-09)

• Allow passing arguments critical/noncritical to only raise an AssertionError if the test is critical [reinhardt]

1.2.2 (2013-04-08)

• Fix to copy all screenshots (not only selenium-screenshot*.png) when mergin test reports

1.2.1 (2013-03-08)

• Revert setting the default zope.testrunner level and support setting it via environment variable ROBOTSUITE_LEVEL (e.g. ROBOTSUITE_LEVEL=2).

1.2.0 (2013-03-08)

• Override test suite __module__ test case id() to support collective.xmltestreport

• Add support for zope.testrunner test suite levels (with default value of 2 to allow easily to run all but robot tests with zope.testrunner

1.1.0 (2013-02-20)

• Change of behavior to not remove existing robot_output.xml, but always just merge new test data into it. robot_output.xml can be reseted by deleting it manually between tests.

1.0.4 (2013-02-19)

• Fix issue in merging of test reports from tests form two or more separate test suites

1.0.3 (2013-02-18)

• Fix the example test to read ZSERVER_PORT -environment variable properly

1.0.2 (2013-02-09)

• Fix to replace spaces in test report folder names

1.0.1 (2013-02-09)

• Add newline before stdout dump on failing tests

1.0.0 (2013-02-08)

• Stop replacing spaces with underscores in testnames

0.9.0 (2012-09-21)

• Added injecting logged errors into captured stdout to be dumped with robot output when a test fails.

0.8.0 (2012-09-13)

• Added support for setting pybot cli-variables from ROBOT_-prefixed environment variables.

0.7.0 (2012-09-11)

• Implemented custom merge for Robot Framework reports to create better concatenated test report and fully support Robot Framework plugin for Jenkins.

• Fixed to copy (selenium-)screenshots into the root test directory to make it easier to publish them in Jenkins.

0.6.0 (2012-08-21)

• Fixed screenshot paths in concatenated report.

• Fixed test names to include suite filename instead of classname.

• Fixed to allow dash in normalized test names.

0.5.0 (2012-07-23)

• Included robotframework test tags as part of tests’ string representation to allow test be filtered with tags (e.g. with zope.testrunner).

• Added creation of concatenated log file.

0.4.0 (2012-06-04)

• Fixed to rename test method of each test after the actual robot framework test run by the test case.

0.3.1 (2012-06-03)

• Updated README.

0.3.0 (2012-06-03)

• Enabled robotframework reports for tests. Each test suite and each test will have their own outputdir, recursively.

0.2.0 (2012-06-02)

• Support for directories as (recursive) suites.

0.1.0 (2012-05-30)

• Proof of concept.