z3c.testsetup 0.4

Defining doctests in Python modules

The doctest file described above was a pure .txt file. By default z3c.testsetup looks for doctests in files with filename extension .txt, .rst and .py. This means, that also doctests in Python modules are found by default as in the following example:

>>> print_file(os.path.join(cavepath, 'doctest08.py'))
|  """
|  Doctests in a Python module
|  ===========================
|
|  We can place doctests also in Python modules.
|
|  :doctest:
|
|  Here the Cave class is defined::
|
|    >>> from z3c.testsetup.tests.othercave.doctest08 import Cave
|    >>> Cave
|    <class 'z3c.testsetup...doctest08.Cave'>
|
|  """
|  class Cave(object):
|      """A Cave.
|
|      A cave has a number::
|
|        >>> hasattr(Cave, 'number')
|        True
|
|      """
|      number = None
|
|      def __init__(self, number):
|          """Create a Cave.
|
|          We have to give a number if we create a cave::
|
|            >>> c = Cave(12)
|            >>> c.number
|            12
|
|          """
|          self.number = number
|

Here we placed the marker string :doctest: into the docstring of the module. Without it, the module would not have been considered a testfile.

Note that you have to import the entities (classes, functions, etc.) from the very same file if you want to use them.

Registering regular unittests from Python modules

z3c.testsetup provides also (limited) support for regular unittest deployments as usually written in Python. An example file could look like this:

>>> print_file(os.path.join(cavepath, 'pythontest1.py'))
|  """
|  Tests with real TestCase objects.
|
|  :unittest:
|
|  """
|
|  import unittest
|
|  class TestTest(unittest.TestCase):
|
|      def setUp(self):
|          pass
|
|      def testFoo(self):
|          self.assertEqual(2, 1+1)
|
|

The module contains a marker :unittest: in its module docstring instead of the :doctest: marker used in the other examples above. It is also the replacement for the formely used :Test-Layer: python marker.

This means, that this file is registered as a regular unittest.

If you use unittests instead of doctests, then you are mainly on your own with setting up and tearing down tests. All this should be done by the test cases themselves.

The only advantage of using z3c.testsetup here is, that those tests are found and run automatically when they provide the marker.

Deprectated/unsupported parameters

The following register_all_tests-parameters are deprecated, starting with z3c.testsetup 0.3:

• filter_func

  and related (ufilter_func, pfilter_func, etc.)

• All testtype specific parameters

  Support for testfile specific parameters (uextensions, fextensions, etc.) is running out and its use deprecated.

Setting up a unittest layer

We can tell z3c.testsetup to use a certain unittest layer using the :layer: marker as in the following example (see tests/othercave/doctest02.txt):

A doctests with layer
=====================
<BLANKLINE>
:doctest:
:layer: z3c.testsetup.tests.othercave.testing.UnitLayer2
<BLANKLINE>
  >>> 1+1
  2

The :doctest: marker was used here as well, because without it the file would not have been detected as a registerable doctest file (we want developers to be explicit about that).

The

:layer: <DOTTED_NAME_OF_LAYER_DEF>

marker then tells, where the testsetup machinery can find the layer definition. It is given in dotted name notation.

How does the layer definition look like? It is defined as regualr Python code:

>>> print open(os.path.join(cavepath, 'testing.py')).read()
import os
...
class UnitLayer1(object):
    """This represents a layer.
    A layer is a way to have common setup and teardown that happens
    once for a whole group of tests.
<BLANKLINE>
    It must be an object with a `setUp` and a `tearDown` method, which
    are run once before or after all the tests applied to a layer
    respectively.
<BLANKLINE>
    Optionally you can additionally define `testSetUp` and
    `testTearDown` methods, which are run before and after each single
    test.
<BLANKLINE>
    This class is not instantiated. Therefore we use classmethods.
    """
<BLANKLINE>
    @classmethod
    def setUp(self):
        """This gets run once for the whole test run, or at most once per
        TestSuite that depends on the layer.
        (The latter can happen if multiple suites depend on the layer
        and the testrunner decides to tear down the layer after first
        suite finishes.)
        """
<BLANKLINE>
    @classmethod
    def tearDown(self):
        """This gets run once for the whole test run, or at most
        once per TestSuite that depends on the layer,
        after all tests in the suite have finished.
        """
<BLANKLINE>
    @classmethod
    def testSetUp(self):
        """This method is run before each single test in the current
        layer. It is optional.
        """
        print "    Running testSetUp of UnitLayer1"
<BLANKLINE>
    @classmethod
    def testTearDown(self):
        """This method is run before each single test in the current
        layer. It is optional.
        """
        print "    Running testTearDown of UnitLayer1"
<BLANKLINE>
class UnitLayer2(UnitLayer1):
    """This Layer inherits ``UnitLayer1``.
<BLANKLINE>
    This way we define nested setups. During test runs the testrunner
    will first call the setup methods of ``UnitTest1`` and then those
    of this class. Handling of teardown-methods will happen the other
    way round.
    """
<BLANKLINE>
    @classmethod
    def setUp(self):
        pass
<BLANKLINE>
    @classmethod
    def testSetUp(self):
        print "    Running testSetUp of UnitLayer2"
<BLANKLINE>
    @classmethod
    def testTearDown(self):
        print "    Running testTearDown of UnitLayer2"

In a layer you can do all the special stuff that is needed to run a certain group of tests properly. Our setup here is special in that we defined a nested one: UnitLayer2 inherits UnitLayer1 so that during test runs the appropriate setup and teardown methods are called (see testrunner output above).

More about test layers can be found at the documentation of testrunner layers API.

Specifying a ZCML file

When it comes to integration or functional tests, we need to specify a ZCML file to which configures the test environment for us. We can do that using the

:zcml-layer: <ZCML-file-name>

marker. It expects a ZCML filename as argument and sets up a ZCML-layered testsuite for us. An example setup might look like so (see tests/othercave/doctest03.txt):

A doctest with a ZCML-layer
===========================

:doctest:
:zcml-layer: ftesting.zcml

  >>> 1+1
  2

Here we say, that the the local file ftesting.zcml should be used as ZCML configuration. As we can see in the above output of testruner, this file is indeed read during test runs and used by a ZCML layer called DefaultZCMLLayer. This layer is in fact only a zope.app.testing.functional.ZCMLLayer.

The ZCML file is looked up in the same directory as the doctest file.

When using the :zcml-layer: marker, the concerned tests are set up via special methods and functions from zope.app.testing. This way you get ‘functional’ or ‘integration’ tests out of the box: in the beginning an empty ZODB db is setup, getRootFolder, sync and other functions are pulled into the test namespace and several things more.

If you want a plain setup instead then use your own layer definition using :layer: and remove the :zcml-layer: marker.

Setting up a functional ZCML layer

Sometimes we want tests to be registered using the FunctionalDocFileSuite function from zope.app.testing.functional (other tests are set up using zope.testing.doctest.DocFileSuite). This function pulls in even more functions into globs, like http (a HTTPCaller instance), wraps your setUp and tearDown methods into ZODB-setups and several things more. See the definition in http://svn.zope.org/zope.app.testing/trunk/src/zope/app/testing/functional.py?view=auto.

This setup needs also a ZCML configuration file, which can be specified via:

:functional-zcml-layer: <ZCML-file-name>

If a functional ZCML layer is specified in a testfile this way, it will override any simple :zcml-layer: or :layer: definition.

An example setup might look like this (see tests/othercave/doctest04.txt):

>>> print_file(os.path.join(cavepath, 'doctest04.txt'))
|  A functional doctest with ZCML-layer
|  ====================================
|
|  :doctest:
|  :functional-zcml-layer: ftesting.zcml
|
|  We didn't define a real environment in ftesting.zcml, but in
|  functional tests certain often needed functions should be available
|  automatically::
|
|    >>> getRootFolder()
|    <zope...folder.Folder object at 0x...>
|

The placeholder between zope and folder was used because the location of the Folder class changed recently. This way we cover setups with old packages as well as recent ones.

Specifying setUp and tearDown methods

We can specify a setUp(test) and tearDown(test) method for the examples in a doctest file, which will be executed once for the whole doctest file. This can be done using:

:setup: <dotted.name.of.callable>
:teardown: <dotted.name.of.callable>

The callables denoted by the dotted names must accept a test parameter which will be the whole test suite of examples in the current doctest file.

An example can be found in doctest05.txt:

>>> print_file(os.path.join(cavepath, 'doctest05.txt'))
|  A doctest with custom setup/teardown functions
|  ==============================================
|
|  :doctest:
|  :setup: z3c.testsetup.tests.othercave.testing.setUp
|  :teardown: z3c.testsetup.tests.othercave.testing.tearDown
|
|    >>> 1+1
|    2
|
|  We make use of a function registered during custom setup::
|
|    >>> myfunc(2)
|    4
|

The setup/teardown functions denoted in the example look like this:

>>> print open(os.path.join(cavepath, 'testing.py'), 'rb').read()
import os
...
def setUp(test):
    print "    Custom setUp for ", test
    # We register a function that will be available during tests.
     test.globs['myfunc'] = lambda x: 2*x
<BLANKLINE>
def tearDown(test):
    print "    Custom tearDown for ", test
    del test.globs['myfunc'] # unregister function
...

As we can see, there is a function myfunc pulled into the namespace of the doctest. We could, however, do arbitrary other things here, set up a relational test database or whatever.

Short setups

Using z3c.testsetup, we can define quite complex testsetups with only two lines of code:

>>> import os
>>> cavepath = os.path.join(os.path.dirname(__file__), 'tests', 'cave')
>>> setupfile = os.path.join(cavepath, 'samplesetup_short0.py')
>>> print open(setupfile).read()
import z3c.testsetup
test_suite = z3c.testsetup.register_all_tests('z3c.testsetup.tests.cave')

We clear the commandline, because all parameters passed to the commandline would otherwise be applied to the examples herein:

>>> import sys
>>> sys.argv = [sys.argv[0],]

This means, that we want to register all tests (doctests and ‘normal’ python tests) from the cave package, whose name we passed in dotted name notation as a string. This is enough information for a testrunner:

>>> defaults = [
...     '--path', cavepath,
...     '--tests-pattern', '^samplesetup_short0$',
...     ]
>>> from z3c.testsetup import testrunner
>>> testrunner.run(defaults)
Running z3c.testsetup.functional.doctesting.FunctionalLayer tests:
  Set up z3c...functional.doctesting.FunctionalLayer in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Running zope.testing.testrunner.layer.UnitTests tests:
  Tear down z3c...functional.doctesting.FunctionalLayer ... not supported
  Running in a subprocess.
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
Total: 4 tests, 0 failures, 0 errors in N.NNN seconds.
False

Of the four tests apparently run, there is one ‘normal’ python test and three doctests, of which two are functional doctests.

Now, we only want to run the doctests in the cave package. A suitable setup is contained in samplesetup_short1.py` in the cave package:

>>> setupfile = os.path.join(cavepath, 'samplesetup_short1.py')
>>> print open(setupfile).read()
import z3c.testsetup
<BLANKLINE>
test_suite = z3c.testsetup.register_doctests('z3c.testsetup.tests.cave')

This means, that we want to register all doctests from the cave package, whose name we passed in dotted name notation as a string. This is enough information for a testrunner:

>>> defaults = [
...     '--path', cavepath,
...     '--tests-pattern', '^samplesetup_short1$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup.functional.doctesting.FunctionalLayer tests:
  Set up z3c.testsetup...doctesting.FunctionalLayer in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Running zope.testing.testrunner.layer.UnitTests tests:
  Tear down z3c.testsetup...doctesting.FunctionalLayer ... not supported
  Running in a subprocess.
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
Total: 3 tests, 0 failures, 0 errors in N.NNN seconds.
False

In the above setup the handled package was given as a string with dotted name notation. We can instead also pass the package itself, if it was loaded before. This results in a slight longer example:

>>> setupfile = os.path.join(cavepath, 'samplesetup_short2.py')
>>> print open(setupfile).read()
import z3c.testsetup
from z3c.testsetup.tests import cave
<BLANKLINE>
test_suite = z3c.testsetup.register_doctests(cave)

Here we register all doctests from the cave module. Let’s start a testrunner with this setup:

>>> defaults = [
...     '--path', cavepath,
...     '--tests-pattern', '^samplesetup_short2$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup.functional.doctesting.FunctionalLayer tests:
  Set up z3c.testsetup...doctesting.FunctionalLayer in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Running zope.testing.testrunner.layer.UnitTests tests:
  Tear down z3c.testsetup...doctesting.FunctionalLayer ... not supported
  Running in a subprocess.
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
Total: 3 tests, 0 failures, 0 errors in N.NNN seconds.
False

Now let’s run a suite of ‘normal’ python unit tests, i.e. tests, that are not doctests. An appropriate setup file might look like this:

>>> setupfile = os.path.join(cavepath, 'samplesetup_short3.py')
>>> print open(setupfile).read()
import z3c.testsetup
from z3c.testsetup.tests import cave
<BLANKLINE>
test_suite = z3c.testsetup.register_pytests(cave)

The only difference to the example before is, that we use register_pytests instead of register_doctests. If we run this setup with the testrunner, one test should be found and executed. This time we pass the -vv option to the testrunner, to get some more information from the run:

>>> defaults = [
...     '--path', cavepath, '-vv',
...     '--tests-pattern', '^samplesetup_short3$',
...     ]
>>> testrunner.run(defaults)
Running tests at level 1
Running zope.testing.testrunner.layer.UnitTests tests:
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Running:
 testFoo (z3c.testsetup.tests.cave.file1.TestTest)
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
Tearing down left over layers:
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
False

Modified short setups

The default settings of test setups might serve for plain testing environments. Especially for functional tests, however, one often want to set some basic values not foreseeable by default. Here some hints, how default settings can be tweaked.

>>> setupfile = os.path.join(cavepath, 'samplesetup_short4.py')
>>> print open(setupfile).read()
import z3c.testsetup
test_suite = z3c.testsetup.register_all_tests(
    'z3c.testsetup.tests.cave',
    zcml_config='sampleftesting.zcml',
    layer_name = 'SampleLayer')

>>> defaults = [
...     '--path', cavepath,
...     '--tests-pattern', '^samplesetup_short4$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup.tests.cave.SampleLayer tests:
  Set up z3c.testsetup.tests.cave.SampleLayer in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Running zope.testing.testrunner.layer.UnitTests tests:
  Tear down z3c.testsetup.tests.cave.SampleLayer ... not supported
  Running in a subprocess.
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
Total: 4 tests, 0 failures, 0 errors in N.NNN seconds.
False

>>> setupfile = os.path.join(cavepath, 'samplesetup_short5.py')
>>> print open(setupfile).read()
import re
from zope.testing import renormalizing
import z3c.testsetup
mychecker = renormalizing.RENormalizing([
    (re.compile('[0-9]*[.][0-9]* seconds'),
     '<SOME NUMBER OF> seconds'),
    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>'),
    ])
test_suite = z3c.testsetup.register_all_tests(
    'z3c.testsetup.tests.cave',
    checker = mychecker,
    extensions = ['.chk',],
    fregexp_list = ['.*checker.*',],
    )

>>> defaults = [
...     '--path', cavepath, '-f',
...     '--tests-pattern', '^samplesetup_short5$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup....doctesting.FunctionalLayer tests:
  Set up z3c.testsetup....doctesting.FunctionalLayer in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Tearing down left over layers:
  Tear down z3c.testsetup....doctesting.FunctionalLayer ... not supported
False

>>> defaults = [
...     '--path', cavepath, '-f',
...     '--tests-pattern', '^samplesetup_short6$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup....doctesting.FunctionalLayer tests:
  Set up z3c.testsetup....doctesting.FunctionalLayer in ... seconds.
<BLANKLINE>
<BLANKLINE>
Failure in test ...checkertest.chk
Failed doctest test for checkertest.chk
  File "...checkertest.chk", line 0
<BLANKLINE>
----------------------------------------------------------------------
File "...checkertest.chk", line 10, in checkertest.chk
Failed example:
    print "%s seconds" % 0.123
Differences (ndiff with -expected +actual):
    - <SOME NUMBER OF> seconds
    + 0.123 seconds
----------------------------------------------------------------------
File "...checkertest.chk", line 15, in checkertest.chk
Failed example:
    print "A memory address at 0x1a0322ff"
Differences (ndiff with -expected +actual):
    - A memory address at <SOME ADDRESS>
    + A memory address at 0x1a0322ff
<BLANKLINE>
  Ran 2 tests with 1 failures and 0 errors in ... seconds.
Tearing down left over layers:
  Tear down z3c.testsetup....doctesting.FunctionalLayer ... not supported
True

>>> setupfile = os.path.join(cavepath, 'samplesetup_short7.py')
>>> print open(setupfile).read()
import os
from zope.testing import renormalizing
import z3c.testsetup
test_suite = z3c.testsetup.register_all_tests(
    'z3c.testsetup.tests.cave',
    extensions = ['.chk',],
    fregexp_list = [':Test-Layer:.*globs.*',],
    globs = {
        'basename' : os.path.basename
    }
    )

>>> defaults = [
...     '--path', cavepath, '-f',
...     '--tests-pattern', '^samplesetup_short7$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup....doctesting.FunctionalLayer tests:
  Set up z3c.testsetup....doctesting.FunctionalLayer in ... seconds.
  Ran 1 tests with 0 failures and 0 errors in ... seconds.
Tearing down left over layers:
  Tear down z3c.testsetup....doctesting.FunctionalLayer ... not supported
False

>>> setupfile = os.path.join(cavepath, 'samplesetup_short8.py')
>>> print open(setupfile).read()
import os
...
test_suite = z3c.testsetup.register_all_tests(
    'z3c.testsetup.tests.cave',
    extensions = ['.chk',],
    fregexp_list = [':Test-Layer:.*globs.*',],
    fglobs = {
        'basename' : os.path.basename
    }
    )

>>> defaults = [
...     '--path', cavepath, '-f',
...     '--tests-pattern', '^samplesetup_short8$',
...     ]
>>> testrunner.run(defaults)
Running z3c.testsetup....doctesting.FunctionalLayer tests:
...
  Ran 1 tests with 0 failures and 0 errors in ... seconds.
...
False

>>> setupfile = os.path.join(cavepath, 'samplesetup_short9.py')
>>> print open(setupfile).read()
import os
...
test_suite = z3c.testsetup.register_all_tests(
    'z3c.testsetup.tests.cave',
    extensions = ['.chk',],
    uregexp_list = [':Test-Layer:.*globs.*',],
    uglobs = {
        'basename' : os.path.basename
    }
    )

>>> defaults = [
...     '--path', cavepath, '-u',
...     '--tests-pattern', '^samplesetup_short9$',
...     ]
>>> testrunner.run(defaults)
Running zope.testing.testrunner.layer.UnitTests tests:
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Tearing down left over layers:
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
False

Extended setups

Let’s have a look at the test setup module samplesetup1 in the cave package:

>>> setupfile = os.path.join(cavepath, 'samplesetup1.py')
>>> print open(setupfile).read()
import unittest
import z3c.testsetup
from z3c.testsetup.tests import cave # The package that contains
                                     # the doctest files
def test_suite():
    suite = unittest.TestSuite()
    suite.addTest( # Add all unittests from `cave`
        z3c.testsetup.UnitDocTestSetup(cave).getTestSuite())
    suite.addTest( # Add all functional tests from `cave`
        z3c.testsetup.FunctionalDocTestSetup(cave).getTestSuite())
    return suite

As we see, there is a unittest setup and a functional test setup initialized. Both collect one kind of tests and feed their collection in the same testsuite (where each kind of tests is setup differently, of course).

Now let’s run a testrunner and see the result. The testrunner will be configured such, that all files named ‘samplesetup1.py’ will be asked to return a testsuite:

>>> defaults = [
...     '--path', cavepath,
...     '--tests-pattern', '^samplesetup1$',
...     ]
>>> from z3c.testsetup import testrunner
>>> testrunner.run(defaults)
Running z3c.testsetup.functional.doctesting.FunctionalLayer tests:
  Set up z3c.testsetup...doctesting.FunctionalLayer in N.NNN seconds.
  Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
Running zope.testing.testrunner.layer.UnitTests tests:
  Tear down z3c.testsetup...doctesting.FunctionalLayer ... not supported
  Running in a subprocess.
  Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
Total: 3 tests, 0 failures, 0 errors in N.NNN seconds.
False

We ran one unittest and two functional tests.

z3c.testsetup and layers

Here we discuss complete testsetups that involve test layers. Test layers are a method to group several tests or test suites in groups.

Layers are a method to do time-consuming setups like initializing databases or whatever only once for a whole bunch of tests. Such you can save time and make your testing setup less error-prone.

See layered_cave/layer.py for examples of working layers. We will use the layers defined there in the following examples.

>>> import os
>>> cavepath = os.path.join(os.path.dirname(__file__), 'tests',
...   'layered_cave')
>>> setupfile = os.path.join(cavepath, 'layeredsetup01.py')
>>> print open(setupfile).read()
from z3c.testsetup import register_all_tests
test_suite = register_all_tests('z3c.testsetup.tests.layered_cave')

>>> testfile = os.path.join(cavepath, 'adoctest.txt')
>>> print open(testfile, 'r').read()
This is a doctest
=================
<BLANKLINE>
This doctest will be applied to a layer.
<BLANKLINE>
:Test-Layer: unit
:Test-Layerdef: z3c.testsetup.tests.layered_cave.layer.UnitLayer1
...

:test-layerdef: <dotted.name.of.layer.definition>

>>> defaults = [
...     '--path', cavepath, '-f',
...     '--tests-pattern', '^layeredsetup01$',
...     ]
>>> testrunner.run(defaults)
Running z3c...DefaultZCMLLayer [/.../bar/ftesting.zcml] tests:
  Set up z3c...DefaultZCMLLayer [/.../bar/ftesting.zcml] in N.NNN seconds.
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
Running z3c...DefaultZCMLLayer [/.../foo/ftesting.zcml] tests:
  Tear down z3c...efaultZCMLLayer [/.../bar/ftesting.zcml] ... not supported
  Running in a subprocess.
  Set up z3c...efaultZCMLLayer [/.../foo/ftesting.zcml] in N.NNN seconds.
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down z3c...efaultZCMLLayer [/.../foo/ftesting.zcml] ... not supported
Running z3c.testsetup.tests.layered_cave.layer.FunctionalLayer1 tests:
  Running in a subprocess.
  Set up z3c...layered_cave.layer.FunctionalLayer1 in N.NNN seconds.
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down z3c...layered_cave.layer.FunctionalLayer1 in N.NNN seconds.
Running z3c.testsetup.tests.layered_cave.layer.UnitLayer1 tests:
  Running in a subprocess.
  Set up z3c.testsetup.tests.layered_cave.layer.UnitLayer1 in N.NNN seconds.
    Running testSetUp of UnitLayer1
    Running testTearDown of UnitLayer1
  Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
  Tear down z3c...layered_cave.layer.UnitLayer1 in N.NNN seconds.
Total: 4 tests, 0 failures, 0 errors in N.NNN seconds.
False

z3c.testsetup.testrunner

With zope.testing 3.7.3 the behaviour of zope.testing.testrunner.run() changed and now exits always with status. See http://svn.zope.org/zope.testing/?rev=99366&view=rev

To run testrunners as part of tests, one now has to use testrunner.run_internal() instead of testrunner.run(). This makes it hard to test with packages, that sometimes use zope.testing >= 3.7.3 but are also used with zope.testing < 3.7.3.

For those cases (i.e. where testrunners are run as part of tests themselves) z3c.testsetup now offers the convenience functions testrunner.run() and testrunner.run_internal() that are wrappers guaranteed to be aliases for the same function of zope.testing: zope.testing.testrunner.run_internal() if it exists or zope.testing.testrunner.run() otherwise.

In doctests you now can use

>>> from z3c.testsetup import testrunner

call testrunner.run() afterwards and it will work, regardless of what version of zope.testing you are using.

This is only interesting for people that run tests, which itself run testrunners (like z3c.testsetup itself in self-tests).

The functions provided by the faked testrunners of z3c.testsetup refer to the same thing:

>>> from z3c.testsetup import testrunner
>>> testrunner.run is testrunner.run_internal
True

If zope.testing >= 3.7.3 is running in background, both functions refer to zope.testing.testrunner.run_internal:

>>> import pkg_resources
>>> info = pkg_resources.get_distribution('zope.testing')
>>> version = tuple(info.version.split('.'))
>>> new_version = (version > ('3', '7', '2'))

>>> from zope.testing.testrunner import run
>>> not new_version or (testrunner.run is not run)
True

Otherwise both functions refer to zope.testing.testrunner.run():

>>> (run is testrunner.run_internal) or new_version
True

Bug fixes

• Made z3c.testsetup selftests work with zope.testing >= 3.7.3. Thanks to Jonathan Ballet for pointing to that problem.

• Ignore *nix hidden test files (i.e. such starting with a dot in filename) by default. Thanks to Jonathan Ballet for patch.

• ZCML files registered via the default layer are now separated from each other, even if they own the same filename. Therefore you can now register a default layer with an ftesting.zcml in one subpackage while having another ftesting.zcml in another package. This was not handled correctly before. Many thanks go to Jonathan Ballet who contributed a patch.

Feature Changes

• Added z3c.testsetup.testrunner that provides wrappers for zope.testing.testrunner``s ``run() and run_internal() functions. Using it, one can make sure that running testrunners inside tests will work regardless of which version of zope.testing is used during testruns.

Bug fixes

• Updated doctest examples to reflect new zope.testing behaviour.

• z3c.testsetup really shouldn’t require zope.app.testing any more. If you use it in an environment without this package, then you cannot register functional tests, which is determined when loading register_all_tests from z3c.testsetup.

• Broken modules are ignored while scanning for tests.

• Modules are not loaded anymore if their source code does not provide a suitable marker string. For this to work, the default checker method isTestModule now expects a martian.scan.ModuleInfo as argument and not a real module. Module infos can be easily created by using module_info_from_dotted_name and module_info_from_package from the martian.scan package.

Feature Changes

• New set of testfile markers:

  • :doctest:

    marks a testfile as a doctest.

  • :unittest:

    marks a testfile as a regular unittest.

  • :layer: dotted.name.to.layer.def

    applies the given layer definition to the tests in the doctest file.

  • :zcml-layer: filename.zcml

    sets up a ZCML layer with the given filename and applies this layer to the doctests in the doctest file.

  • :functional-zcml-layer: filename.zcml

    sets up a ZCML layer with the given filename and applies this layer to the doctests in the doctest file. Furthermore the tests are set up as functional doc tests.

  • :setup: dotted.name.to.setup.function

    applies the setUp function denoted by the dotted name to the tests in the doctest file.

  • :teardown: dotted.name.to.teardown.function

    applies the tearDown function denoted by the dotted name to the tests in the doctests file.

  See the examples in tests/othercave and README.txt to learn more about using these new directives.

  The old :test-layer: marker is still supported but it is deprecated now and will vanish at least with the 0.5 version of z3c.testsetup.

Feature Changes

• An ftesting.zcml in the root of a handled package is now taken as default layer for functional doctests if it exists.

Bug fixes