Hypothesis strategies for Open API / Swagger and GraphQL schemas
Project description
Schemathesis is a tool for testing your web applications built with Open API / Swagger or GraphQL specifications.
It reads the application schema and generates test cases which will ensure that your application is compliant with its schema.
The application under test could be written in any language, the only thing you need is a valid API schema in a supported format.
Supported specification versions:
Swagger 2.0
Open API 3.0.x
GraphQL June 2018
More API specifications will be added in the future.
Built with:
Inspired by wonderful swagger-conformance project.
Docs: https://schemathesis.readthedocs.io/en/stable/
Gitter: https://gitter.im/kiwicom/schemathesis
If you are looking for more information:
An article about Schemathesis by @Stranger6667
A video from FlaskCon 2020 by @hultner
NOTE: I started this project during my work at Kiwi.com. I am grateful to them for all the support they provided to this project during its early days and for the opportunity to evolve Schemathesis independently.
Installation
To install Schemathesis via pip run the following command:
pip install schemathesisUsage
There are two basic ways to use Schemathesis:
CLI is pretty simple to use and requires no coding, in-code approach gives more flexibility.
Command Line Interface
The schemathesis command can be used to perform Schemathesis test cases:
schemathesis run https://example.com/api/swagger.json
If your application requires authorization then you can use --auth option for Basic Auth and --header to specify custom headers to be sent with each request.
To filter your tests by endpoint name, HTTP method, Open API tags or operationId you could use -E, -M, -T, -O options respectively.
CLI supports passing options to hypothesis.settings. All of them are prefixed with --hypothesis-:
schemathesis run --hypothesis-max-examples=1000 https://example.com/api/swagger.jsonTo speed up the testing process Schemathesis provides -w/--workers option for concurrent test execution:
schemathesis run -w 8 https://example.com/api/swagger.jsonIn the example above all tests will be distributed among 8 worker threads.
If you’d like to test your web app (Flask or AioHTTP or FastAPI for example) then there is --app option for you:
schemathesis run --app=importable.path:app /swagger.jsonYou need to specify an importable path to the module where your app instance resides and a variable name after : that points to your app. Note, app factories are not supported. The schema location could be:
A full URL;
An existing filesystem path;
In-app endpoint with schema.
This method is significantly faster for WSGI apps, since it doesn’t involve network.
For the full list of options, run:
schemathesis --help
# Or
schemathesis run --helpDocker
Schemathesis CLI also available as a docker image
docker run kiwicom/schemathesis:stable run http://example.com/schema.jsonTo run it against localhost server add --network=host parameter:
docker run --network="host" kiwicom/schemathesis:stable run http://127.0.0.1/schema.jsonPre-run CLI hook
Sometimes you need to execute custom code before the CLI run, for example setup an environment, register custom string format strategies or modify Schemathesis behavior in runtime you can use --pre-run hook:
schemathesis --pre-run importable.path.to.module run https://example.com/api/swagger.jsonNOTE. This option should be passed before the run part.
The passed value will be processed as an importable Python path, where you can execute your code. An example - https://github.com/schemathesis/schemathesis#custom-string-strategies
Registering custom checks for CLI
To add a new check for the Schemathesis CLI there is a special function
import schemathesis
@schemathesis.register_check
def new_check(response, case):
# some awesome assertions!
passThe registered check should accept a response with requests.Response / schemathesis.utils.WSGIResponse type and case with schemathesis.models.Case type.
After registration, your checks will be available in Schemathesis CLI and you can use them via -c command line option.
schemathesis --pre-run module.with.checks run -c new_check https://example.com/api/swagger.jsonAdditionally, checks may return True to skip the check under certain conditions. For example, you may only want to run checks when the response code is 200.
import schemathesis
@schemathesis.register_check
def conditional_check(response, case):
if response.status_code == 200:
# some awesome assertions!
else:
# check not relevant to this response, skip test
return TrueIn-code
To examine your application with Schemathesis you need to:
Setup & run your application, so it is accessible via the network;
Write a couple of tests in Python;
Run the tests via pytest.
Suppose you have your application running on http://0.0.0.0:8080 and its schema is available at http://0.0.0.0:8080/swagger.json.
A basic test, that will verify that any data, that fit into the schema will not cause any internal server error could look like this:
# test_api.py
import requests
import schemathesis
schema = schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")
@schema.parametrize()
def test_no_server_errors(case):
# `requests` will make an appropriate call under the hood
response = case.call() # use `call_wsgi` if you used `schemathesis.from_wsgi`
# You could use built-in checks
case.validate_response(response)
# Or assert the response manually
assert response.status_code < 500It consists of four main parts:
Schema preparation; schemathesis package provides multiple ways to initialize the schema - from_path, from_dict, from_uri, from_file and from_wsgi.
Test parametrization; @schema.parametrize() generates separate tests for all endpoint/method combination available in the schema.
A network call to the running application; case.call does it.
Verifying a property you’d like to test; In the example, we verify that any app response will not indicate a server-side error (HTTP codes 5xx).
NOTE. Look for from_wsgi usage below
Run the tests:
pytest test_api.pyOther properties that could be tested:
Any call will be processed in <50 ms - you can verify the app performance;
Any unauthorized access will end with 401 HTTP response code;
Each test function should have the case fixture, that represents a single test case.
Important Case attributes:
method - HTTP method
formatted_path - full endpoint path
headers - HTTP headers
query - query parameters
body - request body
You can use them manually in network calls or can convert to a dictionary acceptable by requests.request:
import requests
schema = schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")
@schema.parametrize()
def test_no_server_errors(case):
kwargs = case.as_requests_kwargs()
response = requests.request(**kwargs)For each test, Schemathesis will generate a bunch of random inputs acceptable by the schema. This data could be used to verify that your application works in the way as described in the schema or that schema describes expected behavior.
By default, there will be 100 test cases per endpoint/method combination. To limit the number of examples you could use hypothesis.settings decorator on your test functions:
from hypothesis import settings
@schema.parametrize()
@settings(max_examples=5)
def test_something(client, case):
...To narrow down the scope of the schemathesis tests it is possible to filter by method or endpoint:
@schema.parametrize(method="GET", endpoint="/pet")
def test_no_server_errors(case):
...The acceptable values are regexps or list of regexps (matched with re.search).
WSGI applications support
Schemathesis supports making calls to WSGI-compliant applications instead of real network calls, in this case the test execution will go much faster.
app = Flask("test_app")
@app.route("/schema.json")
def schema():
return {...}
@app.route("/v1/users", methods=["GET"])
def users():
return jsonify([{"name": "Robin"}])
schema = schemathesis.from_wsgi("/schema.json", app)
@schema.parametrize()
def test_no_server_errors(case):
response = case.call_wsgi()
assert response.status_code < 500Explicit examples
If the schema contains parameters examples, then they will be additionally included in the generated cases. Schemathesis supports the use of both OpenAPI example and examples (more information available in the OpenAPI documentation). Note that examples were added in OpenAPI 3, but Schemathesis supports this feature for OpenAPI 2 using x-examples.
paths:
get:
parameters:
- in: body
name: body
required: true
schema: '#/definitions/Pet'
definitions:
Pet:
additionalProperties: false
example:
name: Doggo
properties:
name:
type: string
required:
- name
type: objectWith this Swagger schema example, there will be a case with body {"name": "Doggo"}. Examples handled with example decorator from Hypothesis, more info about its behavior is here.
If you’d like to test only examples provided in the schema, you could utilize --hypothesis-phases=explicit CLI option:
$ schemathesis run --hypothesis-phases=explicit https://example.com/api/swagger.jsonOr add this decorator to your test if you use Schemathesis in your Python tests:
from hypothesis import settings, Phase
...
@schema.parametrize()
@settings(phases=[Phase.explicit])
def test_api(case):
...NOTE. Schemathesis supports examples in individual properties. See below:
...
paths:
/users:
parameters:
- in: query
name: foo
schema:
type: object
properties:
prop1:
type: string
example: prop1 example # SUPPORTED!
post:
requestBody:
content:
application/json:
schema:
type: object
properties:
foo:
type: string
example: bar # SUPPORTED!Direct strategies access
For convenience you can explore the schemas and strategies manually:
>>> import schemathesis
>>> schema = schemathesis.from_uri("http://0.0.0.0:8080/petstore.json")
>>> endpoint = schema["/pet"]["POST"]
>>> strategy = endpoint.as_strategy()
>>> strategy.example()
Case(
path='/pet',
method='POST',
path_parameters={},
headers={},
cookies={},
query={},
body={
'name': '\x15.\x13\U0008f42a',
'photoUrls': ['\x08\U0009f29a', '\U000abfd6\U000427c4', '']
},
form_data={}
)Schema instances implement Mapping protocol.
NOTE. Paths are relative to the schema’s base path (host + basePath in Open API 2.0 and server.url in Open API 3.0):
# your ``basePath`` is ``/api/v1``
>>> schema["/pet"]["POST"] # VALID
>>> schema["/api/v1/pet"]["POST"] # INVALIDLazy loading
If you have a schema that is not available when the tests are collected, for example it is build with tools like apispec and requires an application instance available, then you can parametrize the tests from a pytest fixture.
# test_api.py
import schemathesis
schema = schemathesis.from_pytest_fixture("fixture_name")
@schema.parametrize()
def test_api(case):
...In this case the test body will be used as a sub-test via pytest-subtests library.
NOTE: the used fixture should return a valid schema that could be created via schemathesis.from_dict or other schemathesis.from_ variations.
Extending schemathesis
If you’re looking for a way to extend schemathesis or reuse it in your own application, then runner module might be helpful for you. It can run tests against the given schema URI and will do some simple checks for you.
from schemathesis import runner
events = runner.prepare("http://127.0.0.1:8080/swagger.json")
for event in events:
# do something with eventrunner.prepare creates a generator that yields events of different kinds - BeforeExecution, AfterExecution, etc. They provide a lot of useful information about what happens during tests, but handling of these events is your responsibility. You can take some inspiration from Schemathesis CLI implementation. See full description of events in the source code.
If you want to use Schemathesis CLI with your custom checks, look at this section
The built-in checks list includes the following:
Not a server error. Asserts that response’s status code is less than 500;
Status code conformance. Asserts that response’s status code is listed in the schema;
Content type conformance. Asserts that response’s content type is listed in the schema;
Response schema conformance. Asserts that response’s content conforms to the declared schema;
You can provide your custom checks to the execute function, the check is a callable that accepts one argument of requests.Response type.
from datetime import timedelta
from schemathesis import runner, models
def not_too_long(response, case: models.Case):
assert response.elapsed < timedelta(milliseconds=300)
events = runner.prepare("http://127.0.0.1:8080/swagger.json", checks=[not_too_long])
for event in events:
# do something with eventCustom string strategies
Some string fields could use custom format and validators, e.g. card_number and Luhn algorithm validator.
For such cases it is possible to register custom strategies:
Create hypothesis.strategies.SearchStrategy object
Optionally provide predicate function to filter values
Register it via schemathesis.register_string_format
strategy = strategies.from_regex(r"\A4[0-9]{15}\Z").filter(luhn_validator)
schemathesis.register_string_format("visa_cards", strategy)Unittest support
Schemathesis supports Python’s built-in unittest framework out of the box, you only need to specify strategies for hypothesis.given:
from unittest import TestCase
from hypothesis import given
import schemathesis
schema = schemathesis.from_uri("http://0.0.0.0:8080/petstore.json")
new_pet_strategy = schema["/v2/pet"]["POST"].as_strategy()
class TestSchema(TestCase):
@given(case=new_pet_strategy)
def test_pets(self, case):
response = case.call()
assert response.status_code < 500Schema validation
To avoid obscure and hard to debug errors during test runs Schemathesis validates input schemas for conformance with the relevant spec. If you’d like to disable this behavior use --validate-schema=false in CLI and validate_schema=False argument in loaders.
Local development
First, you need to prepare a virtual environment with poetry. Install poetry (check out the installation guide) and run this command inside the project root:
poetry installFor simpler local development Schemathesis includes a aiohttp-based server with the following endpoints in Swagger 2.0 schema:
/api/success - always returns {"success": true}
/api/failure - always returns 500
/api/slow - always returns {"slow": true} after 250 ms delay
/api/unsatisfiable - parameters for this endpoint are impossible to generate
/api/invalid - invalid parameter definition. Uses int instead of integer
/api/flaky - returns 1/1 ratio of 200/500 responses
/api/multipart - accepts multipart data
/api/teapot - returns 418 status code, that is not listed in the schema
/api/text - returns plain/text responses, which are not declared in the schema
/api/malformed_json - returns malformed JSON with application/json content type header
To start the server:
./test_server.sh 8081It is possible to configure available endpoints via --endpoints option. The value is expected to be a comma separated string with endpoint names (success, failure, slow, etc):
./test_server.sh 8081 --endpoints=success,slowThen you could use CLI against this server:
schemathesis run http://127.0.0.1:8081/schema.yaml
================================== Schemathesis test session starts =================================
platform Linux -- Python 3.7.4, schemathesis-0.12.2, hypothesis-4.39.0, hypothesis_jsonschema-0.9.8
rootdir: /
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/.hypothesis/examples')
Schema location: http://127.0.0.1:8081/schema.yaml
Base URL: http://127.0.0.1:8081
Specification version: Swagger 2.0
collected endpoints: 2
GET /api/slow . [ 50%]
GET /api/success . [100%]
============================================== SUMMARY ==============================================
not_a_server_error 2 / 2 passed PASSED
========================================= 2 passed in 0.29s =========================================Running tests
You could run tests via tox:
tox -p all -oor pytest in your current environment:
pytest test/ -n autoContributing
Any contribution in development, testing or any other area is highly appreciated and useful to the project.
Please, see the CONTRIBUTING.rst file for more details.
Python support
Schemathesis supports Python 3.6, 3.7 and 3.8.
License
The code in this project is licensed under MIT license. By contributing to schemathesis, you agree that your contributions will be licensed under its MIT license.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file schemathesis-2.3.4.tar.gz.
File metadata
- Download URL: schemathesis-2.3.4.tar.gz
- Upload date:
- Size: 88.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.0.10 CPython/3.8.5 Linux/5.4.0-1023-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2f5e91f1edf4457cad9e7bba10fd8ff5c2d55a95bdc028e020a2ec8e03ee9ff9 |
|
| MD5 | 79702f29be6fd3c6a7e8609befbf72e6 |
|
| BLAKE2b-256 | ccdce87180ac790554cd7ba4292967ed74d91169a2bc8362c005287bb9b47043 |
File details
Details for the file schemathesis-2.3.4-py3-none-any.whl.
File metadata
- Download URL: schemathesis-2.3.4-py3-none-any.whl
- Upload date:
- Size: 102.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.0.10 CPython/3.8.5 Linux/5.4.0-1023-azure
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0e569973034a30c881de53813ea61b0c183cbf7763451898d1da9cab408077ad |
|
| MD5 | 321039665f627e8c0e1d3f379a0c1f76 |
|
| BLAKE2b-256 | c6f7974ab6039dd64faf5130c939e4351a4f3008d613b2cc2bbaf4411feb8a73 |
