Skip to main content

A pytest-plugin for updating doctest outputs

Project description

pytest-accept

GitHub Workflow CI Status PyPI Version GitHub License

pytest-accept is a pytest plugin for automatically updating doctest outputs. It runs doctests, observes the generated outputs, and writes them to the doctests' documented outputs.

It's designed for a couple of audiences:

  • Folks who work with doctests, and don't enjoy manually copying & pasting outputs from the pytest error log to their doctests' documented outputs. pytest-accept does the copying & pasting for you.
  • Folks who generally find writing any tests a bit annoying, and prefer to develop by "running the code and seeing if it works". This library aims to make testing a joyful part of that development loop.

pytest-accept is decoupled from the doctests it works with — it can be used with existing doctests, and the doctests it edits are no different from normal doctests.

Jesse, what the?

Here's an example of what pytest-accept does: given a file like add.py containing an incorrect documented output:

def add(x, y):
    """
    Adds two values.

    >>> add(1, 1)
    3

    >>> add("ab", "c")
    'bac'
    """

    return x + y

...running doctests using pytest and passing --accept replaces the existing incorrect values with correct values:

pytest --doctest-modules examples/add.py --accept
diff --git a/examples/add.py b/examples/add.py
index 10a71fd..c2c945f 100644
--- a/examples/add.py
+++ b/examples/add.py
@@ -3,10 +3,10 @@ def add(x, y):
     Adds two values.

     >>> add(1, 1)
-    3
+    2

     >>> add("ab", "c")
-    'bac'
+    'abc'
     """

     return x + y

This style of testing is fairly well-developed in some languages, although still doesn't receive the attention I think it deserves, and historically hasn't had good support in python.

Confusingly, it's referred to "snapshot testing" or "regression testing" or "expect testing" or "literate testing" or "acceptance testing". The best explanation I've seen on this testing style is from @yminsky in a Jane Street Blogpost. @matklad also has an excellent summary in his blog post How to Test.

Installation

pip install pytest-accept

What about pytest tests?

A previous effort in assert_plugin.py attempted to do this for assert statements, and the file contains some notes on the effort. The biggest problem is pytest stops on the first assert failure in each test, which is very limiting. (Whereas pytest can be configured to continue on doctest failures, which this library takes advantage of.)

It's probably possible to change pytest's behavior here, but it's a significant effort on the pytest codebase.

Some alternatives:

  • Use an existing library like pytest-regtest, which offers file snapshot testing (i.e. not inline).
  • We could write a specific function / fixture, like accept(result, "abc"), similar to frameworks like rust's excellent insta (which I developed some features for), or ocaml's ppx_expect.
    • But this has the disadvantage of coupling the test to the plugin: it's not possible to run tests independently of the plugin, or use the plugin on general assert tests. And one of the great elegances of pytest is its deferral to a normal assert statement.
  • Some of this testing feels like writing a notebook and testing that. pytest-notebook fully implements this.

Anything else?

Nothing ground-breaking! Some notes:

  • If a docstring uses escape characters such as \n, python will interpret them as the escape character rather than the literal. Use a raw string to have it interpreted as a literal. e.g. this fails:

    def raw_string():
        """
        >>> "\n"
        '\n'
        """
    

    but succeeds with:

    def raw_string():
    -    """
    +    r"""
        >>> "\n"
        '\n'
    

    Possibly pytest-accept could do more here — e.g. change the format of the docstring. But that would not be trivial to implement, and may be too invasive.

  • The library attempts to confirm the file hasn't changed between the start and end of the test and won't overwrite the file where it detects there's been a change. This can be helpful for workflows where the tests run repeatedly in the background (e.g. using something like watchexec) while a person is working on the file, or when the tests take a long time, maybe because of --pdb. To be doubly careful, passing --accept-copy will cause the plugin to instead create a file named {file}.py.new rather than overwriting the file on any doctest failure.

    • It will overwrite the existing documented values, though these aren't generally useful per se — they're designed to match the generated of the code. The only instances they could be useful is where they've been manual curated (e.g. removing volatile outputs like hashes), and in those cases ideally they can be restored from version control. Or as above, pass --accept-copy to be conservative.
  • This is still fairly early, has mostly been used by me & xarray and there may be some small bugs. Let me know anything at all and I'll attempt to fix them.

  • It currently doesn't affect the printing of test results; the doctests will still print as failures.

    • TODO: A future version could print something about them being fixed.
  • Python's doctest library is imperfect:

    • It can't handle indents, and probably other things.
      • We modify the output to match the doctest format; e.g. with blanklines. If generated output isn't sufficient for the doctest to pass, and there is some form of output that's sufficient, please report as a bug.
    • The syntax for .* is an ellipsis ..., which is also the syntax for continuing a code line, so the beginning of a line must always be specified.
    • The syntax for all the directives is arguably less than aesthetically pleasing.
    • It doesn't have an option for pretty printing, so the test must pretty print itself with pprint(x), which is verbose.
    • It reports line numbers incorrectly in some cases — two docstring lines separated with continuation character \ is counted as one, meaning this library will not have access to the correct line number for doctest inputs and outputs.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pytest-accept-0.1.11.tar.gz (17.5 kB view details)

Uploaded Source

Built Distribution

pytest_accept-0.1.11-py3-none-any.whl (15.5 kB view details)

Uploaded Python 3

File details

Details for the file pytest-accept-0.1.11.tar.gz.

File metadata

  • Download URL: pytest-accept-0.1.11.tar.gz
  • Upload date:
  • Size: 17.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.2.0 CPython/3.8.18 Linux/6.5.0-1025-azure

File hashes

Hashes for pytest-accept-0.1.11.tar.gz
Algorithm Hash digest
SHA256 0f5e69d5fbf48fe1e20330f9005745796a86d441aa89c5f57b95b5f970f98f61
MD5 5e140981dd4797d9d81a471bad30c09e
BLAKE2b-256 a0e26733539ca8bf3066f9621b1d9737a66852fdbdc169df6534ec13ea1b2899

See more details on using hashes here.

File details

Details for the file pytest_accept-0.1.11-py3-none-any.whl.

File metadata

  • Download URL: pytest_accept-0.1.11-py3-none-any.whl
  • Upload date:
  • Size: 15.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.2.0 CPython/3.8.18 Linux/6.5.0-1025-azure

File hashes

Hashes for pytest_accept-0.1.11-py3-none-any.whl
Algorithm Hash digest
SHA256 191f7bdb60a845f14b5eea81087abd9aabc6d4b20c682ce8480b1b72b11384bd
MD5 fd1385552a1be598505dc97be4028b04
BLAKE2b-256 5ef37fa8e9630cfdfe0b4f284f137439edb4d336aba32e4e5020599e50544f8a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page