A script to visualize coverage reports via HTML
Project description
This package produces a nice HTML representation of the coverage data generated by the Zope test runner, zope.testing. It will also highlighting Python syntax if you have enscript installed.
There’s also a script to check for differences in coverage and report any regressions (increases in the number of untested lines).
Detailed Documentation
Test Coverage Reports
The main objective of this package is to convert the text-based coverage output into an HTML coverage report. This is simply done by specifying the directory of the test reports and the desired output directory.
Luckily we already have the text input ready:
>>> import os >>> import z3c.coverage >>> inputDir = os.path.join( ... os.path.split(z3c.coverage.__file__)[0], 'sampleinput')
The output directory has to be created first:
>>> import tempfile >>> outputDir = os.path.join(tempfile.mkdtemp(), 'report')
We can now simply create the coverage report as follows:
>>> from z3c.coverage import coveragereport >>> coveragereport.main((inputDir, outputDir))
Looking at the output directory, we now see several files:
>>> print '\n'.join(sorted(os.listdir(outputDir))) all.html z3c.coverage.__init__.html z3c.coverage.coveragediff.html z3c.coverage.coveragereport.html z3c.coverage.html z3c.html
API Tests
CoverageNode Class
This class represents a node in the source tree. Simple modules are considered leaves and do not have children. Let’s create a node for the z3c namespace first:
>>> z3cNode = coveragereport.CoverageNode()
Before using the API, let’s create a few more nodes and a tree from it:
>>> coverageNode = coveragereport.CoverageNode() >>> z3cNode['coverage'] = coverageNode>>> reportNode = coveragereport.CoverageNode() >>> reportNode._covered, reportNode._total = 40, 134 >>> coverageNode['coveragereport'] = reportNode>>> diffNode = coveragereport.CoverageNode() >>> diffNode._covered, diffNode._total = 128, 128 >>> coverageNode['coveragediff'] = diffNode>>> initNode = coveragereport.CoverageNode() >>> initNode._covered, initNode._total = 0, 0 >>> coverageNode['__init__'] = initNode
Let’s now have a look at the coverage of the z3c namespace:
>>> z3cNode.coverage (168, 262)
We can also ask for the percentile:
>>> z3cNode.percent 64 >>> initNode.percent 100
We can ask for the amount of uncovered lines:
>>> z3cNode.uncovered 94
Finally, we also can get a nice output:
>>> print z3cNode 64% covered (94 of 262 lines uncovered)
index_to_filename() function
Takes an indexed Python path and produces the cover filename for it:
>>> coveragereport.index_to_filename(('z3c', 'coverage', 'coveragereport')) 'z3c.coverage.coveragereport.cover'>>> coveragereport.index_to_filename(()) ''
index_to_nice_name() function
Takes an indexed Python path and produces a nice “human-readable” string:
>>> coveragereport.index_to_nice_name(('z3c', 'coverage', 'coveragereport')) ' coveragereport'>>> coveragereport.index_to_nice_name(()) 'Everything'
index_to_name() function
Takes an indexed Python path and produces a “human-readable” string:
>>> coveragereport.index_to_name(('z3c', 'coverage', 'coveragereport')) 'z3c.coverage.coveragereport'>>> coveragereport.index_to_name(()) 'everything'
percent_to_colour() function
Given a coverage percentage, this function returns a color to represent the coverage:
>>> coveragereport.percent_to_colour(100) 'green' >>> coveragereport.percent_to_colour(92) 'yellow' >>> coveragereport.percent_to_colour(85) 'orange' >>> coveragereport.percent_to_colour(50) 'red'
get_svn_revision() function
Given a path, the function tries to determine the revision number of the file. If it fails, “UNKNOWN” is returned:
>>> path = os.path.split(z3c.coverage.__file__)[0] >>> coveragereport.get_svn_revision(path) != 'UNKNOWN' True>>> coveragereport.get_svn_revision(path + '/__init__.py') 'UNKNOWN'
syntax_highlight() function
This function takes a cover file, converts it to a nicely colored HTML output:
>>> filename = os.path.join( ... os.path.split(z3c.coverage.__file__)[0], '__init__.py')>>> print coveragereport.syntax_highlight(filename) <BLANKLINE> <I><FONT COLOR="#B22222"># Make a package. </FONT></I>
If the highlighing command is not available, no coloration is done:
>>> command_orig = coveragereport.HIGHLIGHT_COMMAND >>> coveragereport.HIGHLIGHT_COMMAND = 'foobar %s'>>> print coveragereport.syntax_highlight(filename) # Make a package. <BLANKLINE>>>> coveragereport.HIGHLIGHT_COMMAND = command_orig
coveragereport.py is a script
For convenience you can download the coveragereport.py module and run it as a script:
>>> import sys >>> sys.argv = ['coveragereport', inputDir, outputDir]>>> script_file = os.path.join( ... z3c.coverage.__path__[0], 'coveragereport.py')>>> execfile(script_file, dict(__name__='__main__'))
Defaults are chosen, when no input and output dir is specified:
>>> def make_coverage_reports_stub(path, report_path): ... print path ... print report_path>>> make_coverage_reports_orig = coveragereport.make_coverage_reports >>> coveragereport.make_coverage_reports = make_coverage_reports_stub>>> sys.argv = ['coveragereport'] >>> coveragereport.main() coverage coverage/reports>>> coveragereport.make_coverage_reports = make_coverage_reports_orig
coveragediff internals
coveragediff is a tool that can be used to compare two directories with coverage reports (such as the ones produced by zope.testing test runner with the --coverage option, or, more generally, the trace module from the Python standard library). coveragediff reports regressions, that is, increases in the number of untested lines of code.
This document describes the internals of coveragediff.py. It also acts as a test suite.
Locating coverage files
The function find_coverage_files looks for plain-text coverage reports in a given directory
>>> import z3c.coverage, os >>> sampleinput_dir = os.path.join(z3c.coverage.__path__[0], 'sampleinput')>>> from z3c.coverage.coveragediff import find_coverage_files >>> for filename in sorted(find_coverage_files(sampleinput_dir)): ... print filename z3c.coverage.__init__.cover z3c.coverage.coveragediff.cover z3c.coverage.coveragereport.cover z3c.coverage.tests.cover
The function filter_coverage_files looks for plain-text coverage reports in a given location that match a set of include and exclude patterns
>>> from z3c.coverage.coveragediff import filter_coverage_files >>> for filename in sorted(filter_coverage_files(sampleinput_dir)): ... print filename z3c.coverage.__init__.cover z3c.coverage.coveragediff.cover z3c.coverage.coveragereport.cover z3c.coverage.tests.cover>>> for filename in sorted(filter_coverage_files(sampleinput_dir, ... include=['diff'])): ... print filename z3c.coverage.coveragediff.cover
The patterns are regular expressions
>>> for filename in sorted(filter_coverage_files(sampleinput_dir, ... exclude=['^z'])): ... print filename
Parsing coverage files
The function count_coverage reads a plain-text coverage reports and returns two numbers: the number of tested code lines and the number of untested code lines.
>>> from z3c.coverage.coveragediff import count_coverage >>> filename = os.path.join(sampleinput_dir, 'z3c.coverage.tests.cover') >>> tested, untested = count_coverage(filename) >>> tested 10 >>> untested 3
Comparing coverage files
The function compare_file reads two coverage reports for the same module and reports a warning if the new file has more untested lines of code
>>> from z3c.coverage.coveragediff import compare_file >>> another_dir = os.path.join(z3c.coverage.__path__[0], 'moresampleinput') >>> old_filename = os.path.join(sampleinput_dir, ... 'z3c.coverage.coveragediff.cover') >>> new_filename = os.path.join(another_dir, ... 'z3c.coverage.coveragediff.cover') >>> compare_file(old_filename, new_filename) z3c.coverage.coveragediff: 36 new lines of untested code
If the number of untested lines is the same or smaller than before, there’s no output
>>> compare_file(new_filename, new_filename) >>> compare_file(new_filename, old_filename)
The function new_file is used to look for untested lines of code in new modules.
>>> from z3c.coverage.coveragediff import new_file >>> new_filename = os.path.join(another_dir, ... 'z3c.coverage.fakenewmodule.cover') >>> new_file(new_filename) z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)
Once again, if there are no untested lines, new_file is quiet
>>> new_filename = os.path.join(another_dir, ... 'z3c.coverage.faketestedmodule.cover') >>> new_file(new_filename)
Comparing directories
compare_dirs ties it all together: you pass in two directory names, you get a bunch of warnings about regressions
>>> from z3c.coverage.coveragediff import compare_dirs >>> compare_dirs(sampleinput_dir, another_dir) z3c.coverage.coveragediff: 36 new lines of untested code z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)
You can pass include and exclude arguments as well
>>> compare_dirs(sampleinput_dir, another_dir, exclude=['[Ff]ake']) z3c.coverage.coveragediff: 36 new lines of untested code>>> compare_dirs(sampleinput_dir, another_dir, include=['d.ff']) z3c.coverage.coveragediff: 36 new lines of untested code
MailSender
The MailSender class is responsible for assembling an RFC-2822 email message and handing it off to an SMTP server.
>>> from z3c.coverage.coveragediff import MailSender >>> mailer = MailSender('smtp.example.com', 25)
Since it wouldn’t be a good idea to actually send emails from the test suite, we’ll use a stub SMTP connection class. Also, let’s hide the real one as an insurance, so that even if our stub fails, the rest of the tests won’t send any real emails:
>>> MailSender.connection_class = None>>> class FakeSMTP(object): ... def __init__(self, host, port): ... print "Connecting to %s:%s" % (host, port) ... def sendmail(self, sender, recipients, body): ... from smtplib import quoteaddr ... print "MAIL FROM:%s" % quoteaddr(sender) ... if isinstance(recipients, basestring): ... recipients = [recipients] ... for recipient in recipients: ... print "RCPT TO:%s" % quoteaddr(recipient) ... print "DATA" ... print body ... print "." ... def quit(self): ... print "QUIT" >>> mailer.connection_class = FakeSMTP
Here’s how you send an email:
>>> mailer.send_email('Some Bot <bot@example.com>', ... 'Maintainer <m@example.com>', ... 'Test coverage regressions', ... 'You broke the tests completely. Have a nice day.') Connecting to smtp.example.com:25 MAIL FROM:<bot@example.com> RCPT TO:<m@example.com> DATA Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Some Bot <bot@example.com> To: Maintainer <m@example.com> Subject: Test coverage regressions <BLANKLINE> You broke the tests completely. Have a nice day. . QUIT
Small utilities
There are several small utility functions like strip, urljoin, matches and filter_files. These are described (and tested) adequately by their doctests.
ReportPrinter
The ReportPrinter class is responsible for formatting the output.
>>> from z3c.coverage.coveragediff import ReportPrinter >>> printer = ReportPrinter() >>> printer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover', ... '3 new untested lines') z3c.coverage.coveragediff: 3 new untested lines >>> printer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover', ... '2 new untested lines') z3c.coverage.coveragereport: 2 new untested lines
Links to web pages
Report printer can also include links to web pages with the coverage reports (e.g. the ones you can get from the coverage tool distributed with z3c.coverage).
>>> printer = ReportPrinter(web_url='http://example.com/coverage/') >>> printer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover', ... '3 new untested lines') z3c.coverage.coveragediff: 3 new untested lines See http://example.com/coverage/z3c.coverage.coveragediff.html <BLANKLINE> >>> printer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover', ... '2 new untested lines') z3c.coverage.coveragereport: 2 new untested lines See http://example.com/coverage/z3c.coverage.coveragereport.html <BLANKLINE>
ReportEmailer
The ReportEmailer class is an alternative to ReportPrinter. It collects warnings and sends them via email.
You pass the basic email parameters (sender, recipient and subject line) to the constructor:
>>> from z3c.coverage.coveragediff import ReportEmailer >>> emailer = ReportEmailer('Some Bot <bot@example.com>', ... 'Maintainer <m@example.com>', ... 'Test coverage regressions')
You add warnings about Python modules by passing the filename of the coverage file and the message
>>> emailer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover', ... '3 new untested lines') >>> emailer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover', ... '2 new untested lines')
Finally you send the email. Since it wouldn’t be a good idea to actually send emails from the test suite, we’ll use a stub MailSender class:
>>> class FakeMailSender(object): ... def send_email(self, from_addr, to_addr, subject, body): ... print "From:", from_addr ... print "To:", to_addr ... print "Subject:", subject ... print "---" ... print body >>> emailer.mailer = FakeMailSender() >>> emailer.send() From: Some Bot <bot@example.com> To: Maintainer <m@example.com> Subject: Test coverage regressions --- z3c.coverage.coveragediff: 3 new untested lines z3c.coverage.coveragereport: 2 new untested lines
Links to web pages
Report emailer can also include links to web pages with the coverage reports (e.g. the ones you can get from the coveragereport tool distributed with z3c.coverage).
>>> emailer = ReportEmailer('Some Bot <bot@example.com>', ... 'Maintainer <m@example.com>', ... 'Test coverage regressions', ... web_url='http://example.com/coverage', ... mailer=FakeMailSender()) >>> emailer.warn('/tmp/coverage/z3c.coverage.coveragediff.cover', ... '3 new untested lines') >>> emailer.warn('/tmp/coverage/z3c.coverage.coveragereport.cover', ... '2 new untested lines') >>> emailer.send() From: Some Bot <bot@example.com> To: Maintainer <m@example.com> Subject: Test coverage regressions --- z3c.coverage.coveragediff: 3 new untested lines See http://example.com/coverage/z3c.coverage.coveragediff.html <BLANKLINE> z3c.coverage.coveragereport: 2 new untested lines See http://example.com/coverage/z3c.coverage.coveragereport.html <BLANKLINE>
Empty reports
Empty reports are not sent out.
>>> emailer = ReportEmailer('Some Bot <bot@example.com>', ... 'Maintainer <m@example.com>', ... 'Test coverage regressions', ... mailer=FakeMailSender()) >>> emailer.send()
Main function
A traditional main function parses command-line arguments and hooks up compare_dirs with the appropriate reporter.
>>> import sys >>> from z3c.coverage.coveragediff import main>>> def run(args): ... try: ... old_stderr = sys.stderr ... sys.argv = args ... sys.stderr = sys.stdout ... try: ... main() ... finally: ... sys.stderr = old_stderr ... except SystemExit, e: ... if e.code: ... print "(returned exit code %s)" % e.code
Help message
>>> run(['coveragediff', '--help']) Usage: coveragediff [options] olddir newdir <BLANKLINE> Options: -h, --help show this help message and exit --include=REGEX only consider files matching REGEX --exclude=REGEX ignore files matching REGEX --email=ADDR send the report to a given email address (only if regressions were found) --from=ADDR set the email sender address --subject=SUBJECT set the email subject --web-url=BASEURL include hyperlinks to HTML-ized coverage reports at a given URL
Missing arguments
>>> run(['coveragediff']) Usage: coveragediff [options] olddir newdir <BLANKLINE> coveragediff: error: wrong number of arguments (returned exit code 2)>>> run(['coveragediff', 'somedir']) Usage: coveragediff [options] olddir newdir <BLANKLINE> coveragediff: error: wrong number of arguments (returned exit code 2)
Excess arguments
>>> run(['coveragediff', 'dir1', 'dir2', 'dir3']) Usage: coveragediff [options] olddir newdir <BLANKLINE> coveragediff: error: wrong number of arguments (returned exit code 2)
Regular run
coveragediff follows the hallowed Unix tradition and does not print any unnecessary output, just the basics
>>> run(['coveragediff', sampleinput_dir, another_dir]) z3c.coverage.coveragediff: 36 new lines of untested code z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)
It means that if you have no coverage regressions in your test suite, the output will be empty
>>> run(['coveragediff', another_dir, another_dir])
Include/exclude patterns
>>> run(['coveragediff', sampleinput_dir, another_dir, ... '--include', 'fake']) z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)>>> run(['coveragediff', sampleinput_dir, another_dir, ... '--exclude', 'fake']) z3c.coverage.coveragediff: 36 new lines of untested code
Links to web pages
If you use coveragereport to produce HTML versions of the plain-text coverage files, and you have those available on the web, you can ask coveragediff to include links to the appropriate modules for convenient copy and paste (or clickety-clicking for those of us who use superior terminal emulators like GNOME Terminal).
>>> run(['coveragediff', sampleinput_dir, another_dir, ... '--web-url', 'http://example.com/coverage']) z3c.coverage.coveragediff: 36 new lines of untested code See http://example.com/coverage/z3c.coverage.coveragediff.html <BLANKLINE> z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13) See http://example.com/coverage/z3c.coverage.fakenewmodule.html <BLANKLINE>
Reports via email
You can ask for the output to be emailed instead of being printed to stdout.
>>> MailSender.connection_class = FakeSMTP >>> run(['coveragediff', sampleinput_dir, another_dir, ... '--email', 'Project List <dev@example.com>', ... '--from', 'Coverage Daemon <root@example.com>']) Connecting to localhost:25 MAIL FROM:<root@example.com> RCPT TO:<dev@example.com> DATA Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Coverage Daemon <root@example.com> To: Project List <dev@example.com> Subject: Unit test coverage regression <BLANKLINE> z3c.coverage.coveragediff: 36 new lines of untested code z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13) . QUIT
coveragediff.py is a script
For convenience you can download the coveragediff.py module and run it as a script
>>> sys.argv = ['coveragediff', sampleinput_dir, another_dir] >>> script_file = os.path.join(z3c.coverage.__path__[0], 'coveragediff.py') >>> execfile(script_file, dict(__name__='__main__')) z3c.coverage.coveragediff: 36 new lines of untested code z3c.coverage.fakenewmodule: new file with 3 lines of untested code (out of 13)
CHANGES
1.2.0 (2010-02-11)
Rename the coverage script to coveragereport, to avoid name clashes with Ned Batchelder’s excellent coverage.py.
1.1.3 (2009-07-24)
Bug: Doctest did not normalize the whitespace in coveragediff.txt. For some reason it passes while testing independently, but when running all tests, it failed.
1.1.2 (2008-04-14)
Bug: When a package path contained anywhere the word “test”, it was ignored from the coverage report. The intended behavior, however, was to ignore files that relate to setting up tests.
Bug: Sort the results of os.listdir() in README.txt to avoid non-deterministic failures.
Bug: The logic for ignoring unit and functional test modules also used to ignore modules and packages called testing.
Change “Unit test coverage” to “Test coverage” in the title – it works perfectly fine for functional tests too.
1.1.1 (2008-01-31)
Bug: When the package was released, the test which tests the availability of an SVN revision number failed. Made the test more reliable.
1.1.0 (2008-01-29)
Feature: The main() coverage report function now accepts the arguments of the script as a function argument, making it easier to configure the script from buildout.
Feature: When the report directory does not exist, the report generator creates it for you.
Feature: Eat your own dog food by creating a buildout that can create coverage reports.
Bug: Improved the test coverage to 100%.
1.0.1 (2007-09-26)
Bug: Fixed meta-data.
1.0.0 (2007-09-26)
First public release.
0.2.1
Feature: Added the --web option to coveragediff.
Feature: Added a test suite.
0.2.0
Feature: Added coveragediff.py.
0.1.0
Initial release of coveragereport.py.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.