z3c.testsetup 0.3

Prerequisites

You need:

- Python 2.4. Rumors are, that also Python 2.5 will do.

• setuptools, available from http://peak.telecommunity.com/DevCenter/setuptools

Other needed packages will be downloaded during installation. Therefore you need an internet connection during installation.

Installation

From the root of the package run:

$ python2.4 bootstrap/bootstrap.py

This will download and install everything to run buildout in the next step. Afterwards an executable script buildout should be available in the freshly created bin/ directory.

Next, fetch all needed packages, install them and create provided scripts:

$ bin/buildout

This should create a test script in bin/.

Running:

$ bin/test

you can test the installed package.

Usage

See README.txt and the other .txt files in the src/z3c/testsetup directory for API documentation.

Note: z3c.testsetup does not fetch all packages you might

need to run your tests automatically by default. Namely the zope.app.testing and zope.component packages are not included if you require only z3c.testsetup in your setup.py.

This is, to make z3c.testsetup compatible with packages that refuse collaboration when those packages are installed. For instance including zope.component in the dependencies would exclude Plone developers from using z3c.testsetup.

Therefore you must include zope.component and zope.app.testing to your requires list in setup.py if you want to use Zope 3 tests. This, however, is done anyway by other packages used in Zope 3 projects and in most cases you do not have to care about this matter.

Basic Example

Before we can find, register and execute tests, we first have to write them down. We already have some ready to use tests available, which can be found in a subpackage:

>>> import os
>>> cavepath = os.path.dirname(__file__)
>>> cavepath = os.path.join(cavepath, 'tests', 'othercave')

In this subpackage there is a simple doctest doctest01.txt (please ignore the pipes on the left):

>>> print_file(os.path.join(cavepath, 'doctest01.txt'))
|  A doctest
|  =========
|
|  :doctest:
|
|  This is a simple doctest.
|
|    >>> 1+1
|    2
|

As we can see, the doctest is marked by a special marker

:doctest:.

This marker tells the testsetup machinery, that this file contains doctest examples that should be registered during test runs. Without this marker, a testfile won’t be registered during tests!

This is the only difference to ‘normal’ doctests here.

Other markers detected by z3c.testsetup are:

• :unittest:

  A replacement for :doctest:, marking a Python module as containing unittests to run. Replaces old Test-Layer: python marker.

• :setup: <dotted.name.of.function>

  Execute the given setup function before running doctests in this file.

• :teardown: <dotted.name.of.function>

  Execute the given teardown function after running doctests in this file.

• :layer: <dotted.name.of.layer.def>

  Use the given layer definition for tests in this file.

• :zcml-layer: <ZCML_filename>

  Use the given ZCML file and run tests in this file on a ZCML layer. Tests are registered using zope.testing.doctest.DocFileSuite.

• :functional-zcml-layer: <ZCML_filename>

  Use the given ZCML file and run tests in this file registered with zope.app.testing.functional.DocFileSuite.

See below for explanations of the respective markers.

Now, that we have a doctest available, we can write a testsetup routine, that collects all tests, registers them and passes them to the testrunner.

We have such a simple testsetup already available:

>>> print open(os.path.join(cavepath, 'simplesetup01.py')).read()
import z3c.testsetup
test_suite = z3c.testsetup.register_all_tests(
    'z3c.testsetup.tests.othercave')

This is all we need in simple cases. We use

register_all_tests(<dotted_pkg_name>)

to tell the setup machinery, where to look for test files. Note, that also files in subpackages will be found, registered and executed, when they are marked approriately.

Let’s start the testrunner and see what it gives:

>>> import sys
>>> sys.argv = [sys.argv[0],]
>>> defaults = [
...     '--path', cavepath,
...     '--tests-pattern', '^simplesetup01$',
...     ]
>>> from zope.testing import testrunner
>>> testrunner.run(defaults)
  Running z3c...functional.layer.DefaultZCMLLayer [ftesting.zcml] tests:
    Set up z3c....layer.DefaultZCMLLayer [ftesting.zcml] in N.NNN seconds.
    Ran 3 tests with 0 failures and 0 errors in N.NNN seconds.
  Running z3c...functional.layer.DefaultZCMLLayer [ftesting2.zcml] tests:
    Tear down z3c...layer.DefaultZCMLLayer [ftesting.zcml] ... not supported
    Running in a subprocess.
    Set up z3c...layer.DefaultZCMLLayer [ftesting2.zcml] in N.NNN seconds.
    Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
    Tear down z3c...layer.DefaultZCMLLayer [ftesting2.zcml] ... not supported
  Running z3c.testsetup.tests.othercave.testing.UnitLayer2 tests:
    Running in a subprocess.
    Set up z3c.testsetup.tests.othercave.testing.UnitLayer1 in N.NNN seconds.
    Set up z3c.testsetup.tests.othercave.testing.UnitLayer2 in N.NNN seconds.
      Running testSetUp of UnitLayer1
      Running testSetUp of UnitLayer2
      Running testTearDown of UnitLayer2
      Running testTearDown of UnitLayer1
    Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
    Tear down z3c...tests.othercave.testing.UnitLayer2 in N.NNN seconds.
    Tear down z3c...tests.othercave.testing.UnitLayer1 in N.NNN seconds.
  Running zope.testing.testrunner.layer.UnitTests tests:
    Running in a subprocess.
    Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
      Custom setUp for  <DocTest doctest05.txt from ... (2 examples)>
      Custom tearDown for  <DocTest doctest05.txt from ... (2 examples)>
    Ran 7 tests with 0 failures and 0 errors in N.NNN seconds.
    Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
  Total: 12 tests, 0 failures, 0 errors in N.NNN seconds.
  False

As we can see, there were regular unittests as well as functional tests run. Some of the unittests used their own layer (UnitLayer1) whose location were printed and the functional tests used different ZCML-files for configuration.

Of course, there were more tests than only the ones defined in doctest01.txt. Let’s have a look at the other stuff.

>>> print_file(os.path.join(cavepath, 'doctest08.py'))
|  """
|  Doctests in a Python module
|  ===========================
|
|  We can place doctests also in Python modules.
|
|  :doctest:
|
|  Here the Cave class is defined::
|
|    >>> from z3c.testsetup.tests.othercave.doctest08 import Cave
|    >>> Cave
|    <class 'z3c.testsetup...doctest08.Cave'>
|
|  """
|  class Cave(object):
|      """A Cave.
|
|      A cave has a number::
|
|        >>> hasattr(Cave, 'number')
|        True
|
|      """
|      number = None
|
|      def __init__(self, number):
|          """Create a Cave.
|
|          We have to give a number if we create a cave::
|
|            >>> c = Cave(12)
|            >>> c.number
|            12
|
|          """
|          self.number = number
|

>>> print_file(os.path.join(cavepath, 'pythontest1.py'))
|  """
|  Tests with real TestCase objects.
|
|  :unittest:
|
|  """
|
|  import unittest
|
|  class TestTest(unittest.TestCase):
|
|      def setUp(self):
|          pass
|
|      def testFoo(self):
|          self.assertEqual(2, 1+1)
|
|

register_all_tests()

The register_all_tests function mentioned above accepts a bunch of keyword parameters:

register_all_tests(pkg_or_dotted_name [, extensions] [, encoding]
                   [, checker] [, globs] [, optionflags]
                   [, setup] [, teardown]
                   [, zcml_config] [, layer_name] [, layer])

where all but the first parameter are keyword paramters and all but the package parameter are optional.

While the extensions parameter determines the set of testfiles to be found, the other paramters tell how to setup single tests.

The last five parameters are only fallbacks, that should better be configured in doctest files themselves via marker strings.

• extensions:

  a list of filename extensions to be considered during doctest search. Default value for doctests is [‘.txt’, ‘.rst’, ‘.py’]. Python tests are not touched by this (they have to be regular Python modules with ‘.py’ extension).

  If we want to register .foo files, we can do so:

  >>> import z3c.testsetup
  >>> test_suite = z3c.testsetup.register_all_tests(
  ...     'z3c.testsetup.tests.cave',
  ...     extensions=['.foo'])
  >>> suite = test_suite()
  >>> get_basenames_from_suite(suite)
  ['file1.py', 'notatest1.foo', 'notatest1.foo']

  Note, that only files that contain an appropriate marker are found, regardless of the filename extension.

• encoding:

  the encoding of testfiles. ‘utf-8’ by default. Setting this to None means using the default value. We’ve hidden one doctest file, that contains umlauts. If we set the encoding to ascii, we get an error:

  >>> test_suite = z3c.testsetup.register_all_tests(
  ...     'z3c.testsetup.tests.cave',
  ...     encoding='ascii')
  >>> suite = test_suite()
  Traceback (most recent call last):
  ...
  UnicodeDecodeError: 'ascii' codec can't decode ...: ordinal
  not in range(128)

  While using ‘latin-1’ will work:

  >>> test_suite = z3c.testsetup.register_all_tests(
  ...     'z3c.testsetup.tests.cave',
  ...     encoding='latin-1')
  >>> suite = test_suite()

  No traceback here.

  You can always overwrite an encoding setting for a certain file by following PEP 0263 ( http://www.python.org/dev/peps/pep-0263/ ).

• checker:

  An output checker for doctests. None by default. A typical output checker can be created like this:

  >>> import re
  >>> from zope.testing import renormalizing
  >>> mychecker = renormalizing.RENormalizing([
  ...    (re.compile('[0-9]*[.][0-9]* seconds'),
  ...     '<SOME NUMBER OF> seconds'),
  ...    (re.compile('at 0x[0-9a-f]+'), 'at <SOME ADDRESS>'),
  ... ])

  This would match for example output like 0.123 seconds if you write in your doctest:

  <SOME NUBMER OF> seconds

  Please see testrunner.txt for examples of usage.

  Checkers are applied to functional doctests only!

• globs:

  A dictionary of things that should be available immediately (without imports) during tests. Default is an empty dict, which might be populated by appropriate layers (see below). ZCML layers for example get you the getRootFolder method automatically.

  This parameter is a fallback which can be overriden by testfile markers specifying a certain layer (see below).

  The globs parameter applies only to doctests.

• optionflags:

  Optionflags influence the behaviour of the testrunner. They are logically or’d so that you can add them arithmetically. See

  http://svn.zope.org/zope.testing/trunk/src/zope/testing/doctest.py

  for details.

• setup:

  A callable that takes a test argument and is executed before every single doctest.

  The default function does nothing.

  This parameter is a fallback which can be overriden by testfile markers specifying a certain layer (see below).

  Specifying setup functions in a layer is also the recommended way.

• teardown:

  The equivalent to setup.

  The default function runs

  zope.testing.cleanup.cleanUp()

  unless overriden by a layer.

  Specifying teardown functions in a layer is also the recommended way.

• zcml_config:

  A filepath of a ZCML file which is registered with functional doctests. In the ZCML file you can for example register principals (users) usable by functional doctests.

  By default any ftesting.zcml file from the root of the given package is taken. If this does not exist, an empty ZCML file of the z3c.testsetup package is used (ftesting.zcml).

  This parameter has no effect, if also a layer parameter is given or a docfile specifies its own layer/ZCML config (see below).

  This is a fallback parameter. Use of docfile specific layer markers is recommended.

• layer_name:

  You can name your layer, to distinguish different setups of functional doctests. The layer name can be an arbitrary string.

  This parameter has no effect, if also a layer parameter is given or a docfile specifies its own layer/ZCML config (see below).

  This is a fallback parameter. Use of docfile specific layer markers is recommended.

• layer:

  You can register a ZCML layer yourself and pass it as the layer parameter. If you only have a filepath to the according ZCML file, use the zcml_config paramter instead.

  This parameter overrides any zcml_config and layer_name parameter.

  This is a fallback parameter and has no effect for docfiles specifying their own layer or ZCML config.

Layers and setup/teardown functions

Starting with z3c.testsetup 0.3 there is first reasonable support for setting up layers per testfile. This way you can easily create setup-functions that are only run before/after certain tests.

Overall, use of layers is the recommended way from now on.

A doctests with layer
=====================
<BLANKLINE>
:doctest:
:layer: z3c.testsetup.tests.othercave.testing.UnitLayer2
<BLANKLINE>
  >>> 1+1
  2

>>> print open(os.path.join(cavepath, 'testing.py')).read()
import os
...
class UnitLayer1(object):
    """This represents a layer.
    A layer is a way to have common setup and teardown that happens
    once for a whole group of tests.
<BLANKLINE>
    It must be an object with a `setUp` and a `tearDown` method, which
    are run once before or after all the tests applied to a layer
    respectively.
<BLANKLINE>
    Optionally you can additionally define `testSetUp` and
    `testTearDown` methods, which are run before and after each single
    test.
<BLANKLINE>
    This class is not instantiated. Therefore we use classmethods.
    """
<BLANKLINE>
    @classmethod
    def setUp(self):
        """This gets run once for the whole test run, or at most once per
        TestSuite that depends on the layer.
        (The latter can happen if multiple suites depend on the layer
        and the testrunner decides to tear down the layer after first
        suite finishes.)
        """
<BLANKLINE>
    @classmethod
    def tearDown(self):
        """This gets run once for the whole test run, or at most
        once per TestSuite that depends on the layer,
        after all tests in the suite have finished.
        """
<BLANKLINE>
    @classmethod
    def testSetUp(self):
        """This method is run before each single test in the current
        layer. It is optional.
        """
        print "    Running testSetUp of UnitLayer1"
<BLANKLINE>
    @classmethod
    def testTearDown(self):
        """This method is run before each single test in the current
        layer. It is optional.
        """
        print "    Running testTearDown of UnitLayer1"
<BLANKLINE>
class UnitLayer2(UnitLayer1):
    """This Layer inherits ``UnitLayer1``.
<BLANKLINE>
    This way we define nested setups. During test runs the testrunner
    will first call the setup methods of ``UnitTest1`` and then those
    of this class. Handling of teardown-methods will happen the other
    way round.
    """
<BLANKLINE>
    @classmethod
    def setUp(self):
        pass
<BLANKLINE>
    @classmethod
    def testSetUp(self):
        print "    Running testSetUp of UnitLayer2"
<BLANKLINE>
    @classmethod
    def testTearDown(self):
        print "    Running testTearDown of UnitLayer2"

A doctest with a ZCML-layer
===========================

:doctest:
:zcml-layer: ftesting.zcml

  >>> 1+1
  2

:functional-zcml-layer: <ZCML-file-name>

>>> print_file(os.path.join(cavepath, 'doctest04.txt'))
|  A functional doctest with ZCML-layer
|  ====================================
|
|  :doctest:
|  :functional-zcml-layer: ftesting.zcml
|
|  We didn't define a real environment in ftesting.zcml, but in
|  functional tests certain often needed functions should be available
|  automatically::
|
|    >>> getRootFolder()
|    <zope...folder.Folder object at 0x...>
|

:setup: <dotted.name.of.callable>
:teardown: <dotted.name.of.callable>

>>> print_file(os.path.join(cavepath, 'doctest05.txt'))
|  A doctest with custom setup/teardown functions
|  ==============================================
|
|  :doctest:
|  :setup: z3c.testsetup.tests.othercave.testing.setUp
|  :teardown: z3c.testsetup.tests.othercave.testing.tearDown
|
|    >>> 1+1
|    2
|
|  We make use of a function registered during custom setup::
|
|    >>> myfunc(2)
|    4
|

>>> print open(os.path.join(cavepath, 'testing.py'), 'rb').read()
import os
...
def setUp(test):
    print "    Custom setUp for ", test
    # We register a function that will be available during tests.
     test.globs['myfunc'] = lambda x: 2*x
<BLANKLINE>
def tearDown(test):
    print "    Custom tearDown for ", test
    del test.globs['myfunc'] # unregister function
...

How to upgrade from z3c.testsetup < 0.3

With the 0.3 release of z3c.testsetup the set of valid marker strings changed, introducing support for file-dependent setups, layers, etc.

If you still mark your testfiles with the :Test-Layer: marker, update your testfiles as follows:

• :Test-Layer: unit

  Change to: :doctest:

• :Test-Layer: python

  Change to: :unittest:

• :Test-Layer: functional

  Change to: :functional-zcml-layer: <ZCML-FILE>

  The ZCML file must explicitly be given.

If you used custom setups passed to register_all_tests, consider declaring those setup/teardown functions in the appropriate doctest files using :setup: and :teardown:.

You might also get better structured test suites when using the new layer markers :layer:, :zcml-layer: and functional-zcml-layer:.

Try to get rid of all parameters passed to register_all_tests except those, mentioned in README.txt.

0.3 (2009-02-23)

0.2.2 (2008-02-29)

Bug fixes

• z3c.testsetup now does not require zope.component nor zope.app.testing for usage in other packages. You must take care, that those packages are available during tests, for example by adding those packages to your setup.py.

0.2.1 (2008-02-18)

Bug fixes

• Fix faulty upload egg.

0.2 (2008-02-17)

0.1 (2008-02-15)

Feature changes

• Initial Release