Hypothesis strategies for Open API / Swagger schemas
Project description
Schemathesis is a tool for testing your web applications built with Open API / Swagger specifications.
It reads the application schema and generates test cases which will ensure that your application is compliant with its schema.
The application under test could be written in any language, the only thing you need is a valid API schema in a supported format.
Supported specification versions:
Swagger 2.0
Open API 3.0.x
More API specifications will be added in the future.
Built with:
Inspired by wonderful swagger-conformance project.
Installation
To install Schemathesis via pip run the following command:
pip install schemathesisOptionally you could install requests for convenient HTTP calls.
Usage
To examine your application with Schemathesis you need to:
Setup & run your application, so it is accessible via the network;
Write a couple of tests in Python;
Run the tests via pytest.
Suppose you have your application running on http://0.0.0.0:8080 and its schema is available at http://0.0.0.0:8080/swagger.json.
A basic test, that will verify that any data, that fit into the schema will not cause any internal server error could look like this:
# test_api.py
import requests
import schemathesis
BASE_URL = "http://0.0.0.0:8080"
schema = schemathesis.from_uri(f"{BASE_URL}/swagger.json")
@schema.parametrize()
def test_no_server_errors(case):
response = requests.request(
case.method,
f"{BASE_URL}{case.formatted_path}",
headers=case.headers,
params=case.query,
json=case.body
)
assert response.status_code < 500It consists of four main parts:
Schema preparation; schemathesis package provides multiple ways to initialize the schema - from_path, from_dict, from_uri, from_file.
Test parametrization; @schema.parametrize() generates separate tests for all endpoint/method combination available in the schema.
A network call to the running application; requests will do the job, for example.
Verifying a property you’d like to test; In the example, we verify that any app response will not indicate a server-side error (HTTP codes 5xx).
Run the tests:
pytest test_api.pyOther properties that could be tested:
Any call will be processed in <50 ms - you can verify the app performance;
Any unauthorized access will end with 401 HTTP response code;
Each test function should have the case fixture, that represents a single test case.
Important Case attributes:
method - HTTP method
formatted_path - full endpoint path
headers - HTTP headers
query - query parameters
body - request body
For each test, Schemathesis will generate a bunch of random inputs acceptable by the schema. This data could be used to verify that your application works in the way as described in the schema or that schema describes expected behavior.
By default, there will be 100 test cases per endpoint/method combination. To limit the number of examples you could use hypothesis.settings decorator on your test functions:
from hypothesis import settings
@settings(max_examples=5)
def test_something(client, case):
...To narrow down the scope of the schemathesis tests it is possible to filter by method or endpoint:
@schema.parametrize(method="GET", endpoint="/pet")
def test_no_server_errors(case):
...The acceptable values are regexps or list of regexps (matched with re.search).
Explicit examples
If the schema contains parameters examples, then they will be additionally included in the generated cases.
paths:
get:
parameters:
- in: body
name: body
required: true
schema: '#/definitions/Pet'
definitions:
Pet:
additionalProperties: false
example:
name: Doggo
properties:
name:
type: string
required:
- name
type: objectWith this Swagger schema example, there will be a case with body {"name": "Doggo"}. Examples handled with example decorator from Hypothesis, more info about its behavior is here.
NOTE. Schemathesis supports only examples in parameters at the moment, examples of individual properties are not supported.
Lazy loading
If you have a schema that is not available when the tests are collected, for example it is build with tools like apispec and requires an application instance available, then you can parametrize the tests from a pytest fixture.
# test_api.py
import schemathesis
schema = schemathesis.from_pytest_fixture("fixture_name")
@schema.parametrize()
def test_api(case):
...In this case the test body will be used as a sub-test via pytest-subtests library.
NOTE: the used fixture should return a valid schema that could be created via schemathesis.from_dict or other schemathesis.from_ variations.
Extending schemathesis
If you’re looking for a way to extend schemathesis or reuse it in your own application, then runner module might be helpful for you. It can run tests against the given schema URI and will do some simple checks for you.
from schemathesis import runner
runner.execute("http://127.0.0.1:8080/swagger.json")The built-in checks list includes the following:
Not a server error. Asserts that response’s status code is less than 500;
You can provide your custom checks to the execute function, the check is a callable that accepts one argument of requests.Response type.
from datetime import timedelta
from schemathesis import runner
def not_too_long(response):
assert response.elapsed < timedelta(milliseconds=300)
runner.execute("http://127.0.0.1:8080/swagger.json", checks=[not_too_long])Documentation
For the full documentation, please see https://schemathesis.readthedocs.io/en/latest/ (WIP)
Or you can look at the docs/ directory in the repository.
Contributing
Any contribution in development, testing or any other area is highly appreciated and useful to the project.
Please, see the CONTRIBUTING.rst file for more details.
Python support
Schemathesis supports Python 3.6, 3.7 and 3.8.
License
The code in this project is licensed under MIT license. By contributing to schemathesis, you agree that your contributions will be licensed under its MIT license.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file schemathesis-0.7.3-py3-none-any.whl.
File metadata
- Download URL: schemathesis-0.7.3-py3-none-any.whl
- Upload date:
- Size: 15.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.7.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 100da501a81107e134fef24a6fb1056370916854a113c2bd724db85e47e4d529 |
|
| MD5 | 766b87dd493cc1453187ece9a700da7b |
|
| BLAKE2b-256 | 514a554adb193aa19df56c00782ba4bc44bd9c2e77955f2bc8926d04578b5227 |
