testimony 1.0.2

Advantages

1. Avoids a separate test case management tool to document test cases by using the python automation framework for the same

2. Enforces test cases to follow a defined standard

3. Runs in integration with tools like Travis to get a report as and when you check in code

4. Saves a lot of time from the conventional way of writing test cases using a test management tool

Installation

You could install testimony from PyPI using pip:

pip install testimony

Requirements

1) Install testimony following the steps on the installation section

2) Pre-requisites

Expected Docstring format:

"""@Test: Log in as a valid user

@Feature: Login

@Setup: Navigate to abc.com

@Steps:

1. Launch the url
2. Log in with valid user credentials

@Assert: Log in successful

@BZ: #1234567

@Status: Manual (REMOVE this field once automated)

@Tags: T1, T2, T3

"""

3) Optional color formatting - If termcolor package is installed, output will be printed in colored text

Usage:

$ testimony -h
usage: testimony [-h] [-n] [-t TAGS] REPORT PATH [PATH ...]

Inspects and report on the Python test cases.

positional arguments:
    REPORT                report type, possible values: print, summary,
                      validate_docstring, bugs, manual, auto, tags
    PATH                  a list of paths to look for tests cases

    optional arguments:
    -h, --help            show this help message and exit
    -j, --json            JSON output
    -n, --nocolor         Do not use color option
    -t [TAGS [TAGS ...]], --tags [TAGS [TAGS ...]]
                      space separated tags to search.  Note: Always run this
                      only in the root of the project where test cases are
                      stored

$ testimony print /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Scanning test_sample.py...

TC 1
Feture: Login - Positive
Test: Login with right credentials
Steps:
    1. Login to the application with valid credentials
Assert: Login is successful
Bug: 123456
Statues: Manual

TC 2
test_positive_login_2: Docstring missing. Please update.

TC 3
Feature: Login - Positive
Test: Login with Latin credentials
Steps:
    1. Login to the application with valid Latin credentials
Assert: Login is successful

TC 4
Feature: Login - Positive
Test: Login with Credentials having special characters
Steps:
    1. Login to the application with valid credentials having
    special characters
Assert: Activation key is created
Status: Manual

TC 5
Steps:
    1. Login to the application with invalid credentials
BZ: 123456
Status: Manual

$ echo $?
255

$ testimony summary /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Total Number of test cases:      4
Total Number of automated cases: 2
Total Number of manual cases:    2
Test cases with no docstrings:   1


$ echo $?
255

$ testimony validate_docstring /home/testimony/tests/

Fetching Test Path /home/testimony/tests/

Scanning test_sample.py...
test_positive_login_1: Invalid DocString: Feture: Login - Positive
test_positive_login_1: Invalid DocString: Bug: 123456
test_positive_login_1: Invalid DocString: Statues: Manual
test_positive_login_1: Need feature, test and assert at the minimum
test_positive_login_2: Docstring missing. Please update.
test_negative_login_5: Need feature, test and assert at the minimum
Total Number of invalid docstrings:  3
Test cases with no docstrings:   1
Test cases missing minimal docstrings:  2

$ echo $?
255

$ testimony bugs /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...

Total Number of test cases affected by bugs: 1

Bug list:
123456

$ echo $?
255

 $ testimony manual /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...
Feature: Login - Positive
Test: Login with Credentials having special characters
Steps:
    1. Login to the application with valid credentials having
    special characters
Assert: Activation key is created
Status: Manual
Steps:
    1. Login to the application with invalid credentials
BZ: 123456
Status: Manual

$ echo $?
255

$ testimony auto /home/testimony/tests/

Fetching Test Path /home/estimony/tests/

Scanning test_sample.py...
Feture: Login - Positive
Test: Login with right credentials
Steps:
    1. Login to the application with valid credentials
Assert: Login is successful
Bug: 123456
Statues: Manual
Feature: Login - Positive
Test: Login with Latin credentials
Steps:
    1. Login to the application with valid Latin credentials
Assert: Login is successful

$ echo $?
255

$ testimony tags tests/ --tag t1
    ['tests.test_sample.Testsample1.test_positive_login_1',
     'tests.test_sample.Testsample1.test_positive_login_3']

$ testimony tags tests/ --tag t1 t2
    ['tests.test_sample.Testsample1.test_positive_login_1',
     'tests.test_sample.Testsample1.test_positive_login_3',
     'tests.test_sample.Testsample1.test_negative_login_5']

Success scenario in which testimony returns 0

$ testimony validate_docstring /home/tests/ui/sample/

Fetching Test Path home/tests/ui/sample/

Scanning test_activationkey.py...
Total Number of invalid docstrings:  0
Test cases with no docstrings:   0
Test cases missing minimal docstrings:  0

$ echo $?
0

Having termcolor installed, testimony produces colored output by default. It can be disabled by:

$ testimony auto /home/apple/tests/login/ --nocolor

(or)

$ testimony auto /home/apple/tests/login/ -n

Testimony supports json output format to integrate with other applications easily. This can be done by adding –json or -j to any of the testimony commands as shown below:

$ testimony summary --json tests/
    [{"auto_count": 2, "manual_count": 2, "auto_percent": 50.0, "no_docstring": 1, "path": "tests/", "tc_count": 4, "manual_percent": 50.0}]

    $ testimony summary -j tests/
    [{"auto_count": 2, "manual_count": 2, "auto_percent": 50.0, "no_docstring": 1, "path": "tests/", "tc_count": 4, "manual_percent": 50.0}]

Known Issues

None

Version History

• Version 1.0.2

- Major code refactor for modularizing the code
- Added Tags support
- If @Test is not present, the first line of docstring will be used instead

• Version 1.0.0

- json support now incorporated

• Version 0.3.0

- Bug fix: Manual vs. automated test count is wrong when the test cases are written with "status" tag vs. "Status"

• Version 0.2.0

- fix to check the tests starting with test_ rather than just test
- Testimony will return error code when docstrings are missing, incorrect docstrings found, minimal docstrings not present
- Make validate_docstring return a 0 success return code if no errors are found
- Organized Constants
- Now testimony accepts --nocolor or --n argument to avoid color output
- testimony will now not error out if termcolor is not installed.
- Make termcolor an optional dependency
- Add Travis configuration to automatically run pep8 when testimony is updated
- Get tests from subfolders of the given path

• Version 0.1.0

- Initial Release

Author

This software is developed by Suresh Thirugn.

Contributors