testimony 1.0.0

Advantages

1. Avoids a separate test case management tool to document test cases by using the python automation framework for the same

2. Enforces test cases to follow a defined standard

3. Runs in integration with tools like Travis to get a report as and when you check in code

4. Saves a lot of time from the conventional way of writing test cases using a test management tool

Installation

You could install testimony from PyPI using pip:

pip install testimony

Requirements

1) Install testimony following the steps on the installation section

2) Pre-requisites

Expected Docstring format:

"""
@Feature: Login
@Test: Log in as a valid user
@Setup: Navigate to abc.com
@Steps:
 1.  Launch the url
 2.  Log in with valid user credentials
@Assert: Log in successful
@BZ: #1234567
@Status: Manual (REMOVE this field once automated)
"""

3) Optional color formatting - If termcolor package is installed, output will be printed in colored text

Usage:

$ testimony -h
usage: testimony [-h] [-n] REPORT PATH [PATH ...]

Inspects and report on the Python test cases.

positional arguments:
    REPORT         report type, possible values: print, summary,
               validate_docstring, bugs, manual, auto
    PATH           a list of paths to look for tests cases

    optional arguments:
    -h, --help     show this help message and exit
    -j, --json     JSON output
    -n, --nocolor  Do not use color option

$ testimony print /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Scanning test_sample.py...

TC 1
Feture: Login - Positive
Test: Login with right credentials
Steps:
    1. Login to the application with valid credentials
Assert: Login is successful
Bug: 123456
Statues: Manual

TC 2
test_positive_login_2: Docstring missing. Please update.

TC 3
Feature: Login - Positive
Test: Login with Latin credentials
Steps:
    1. Login to the application with valid Latin credentials
Assert: Login is successful

TC 4
Feature: Login - Positive
Test: Login with Credentials having special characters
Steps:
    1. Login to the application with valid credentials having
    special characters
Assert: Activation key is created
Status: Manual

TC 5
Steps:
    1. Login to the application with invalid credentials
BZ: 123456
Status: Manual

$ echo $?
255

$ testimony summary /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Total Number of test cases:      4
Total Number of automated cases: 2
Total Number of manual cases:    2
Test cases with no docstrings:   1


$ echo $?
255

$ testimony validate_docstring /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Scanning test_sample.py...
test_positive_login_1: Invalid DocString: Feture: Login - Positive
test_positive_login_1: Invalid DocString: Bug: 123456
test_positive_login_1: Invalid DocString: Statues: Manual
test_positive_login_1: Need feature, test and assert at the minimum
test_positive_login_2: Docstring missing. Please update.
test_negative_login_5: Need feature, test and assert at the minimum
Total Number of invalid docstrings:  3
Test cases with no docstrings:   1
Test cases missing minimal docstrings:  2

$ echo $?
255

$ testimony bugs /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...

Total Number of test cases affected by bugs: 1

Bug list:
123456

$ echo $?
255

 $ testimony manual /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...
Feature: Login - Positive
Test: Login with Credentials having special characters
Steps:
    1. Login to the application with valid credentials having
    special characters
Assert: Activation key is created
Status: Manual
Steps:
    1. Login to the application with invalid credentials
BZ: 123456
Status: Manual

$ echo $?
255

$ testimony auto /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...
Feture: Login - Positive
Test: Login with right credentials
Steps:
    1. Login to the application with valid credentials
Assert: Login is successful
Bug: 123456
Statues: Manual
Feature: Login - Positive
Test: Login with Latin credentials
Steps:
    1. Login to the application with valid Latin credentials
Assert: Login is successful

$ echo $?
255

Success scenario in which testimony returns 0

$ testimony validate_docstring /home/tests/ui/sample/

Fetching Test Path home/tests/ui/sample/

Scanning test_activationkey.py...
Total Number of invalid docstrings:  0
Test cases with no docstrings:   0
Test cases missing minimal docstrings:  0

$ echo $?
0

Having termcolor installed, testimony produces colored output by default. It can be disabled by:

$ testimony auto /home/apple/tests/login/ --nocolor

(or)

$ testimony auto /home/apple/tests/login/ -n

Testimony supports json output format to integrate with other applications easily. This can be done by adding –json or -j to any of the testimony commands as shown below:

$ testimony summary --json tests/
    [{"auto_count": 2, "manual_count": 2, "auto_percent": 50.0, "no_docstring": 1, "path": "tests/", "tc_count": 4, "manual_percent": 50.0}]

    $ testimony summary -j tests/
    [{"auto_count": 2, "manual_count": 2, "auto_percent": 50.0, "no_docstring": 1, "path": "tests/", "tc_count": 4, "manual_percent": 50.0}]

Known Issues

None

Version History

• Version 1.0.0

- json support now incorporated

• Version 0.3.0

- Bug fix: Manual vs. automated test count is wrong when the test cases are written with "status" tag vs. "Status"

• Version 0.2.0

- fix to check the tests starting with test_ rather than just test
- Testimony will return error code when docstrings are missing, incorrect docstrings found, minimal docstrings not present
- Make validate_docstring return a 0 success return code if no errors are found
- Organized Constants
- Now testimony accepts --nocolor or --n argument to avoid color output
- testimony will now not error out if termcolor is not installed.
- Make termcolor an optional dependency
- Add Travis configuration to automatically run pep8 when testimony is updated
- Get tests from subfolders of the given path

• Version 0.1.0

- Initial Release

Author

This software is developed by Suresh Thirugn.

Contributors