hypofuzz

Adaptive fuzzing of Hypothesis tests.

Property-based approaches help you to write better tests which find more bugs, but don't have great ways to exchange much more CPU time for more bugs. The goal of this project is to bring togther the best parts of fuzzing and PBT.

Motivation

You can run a traditional fuzzer like AFL on Hypothesis tests to get basic coverage guidance.

• This does actually work, which is pretty cool
• It's very slow though, and often fails to parse the bytes into an example
• Installing, configuring, and connecting all the parts is a pain
• Also assumes one fuzz target per core, which doesn't scale very far

Alternatively, you can just run Hypothesis with a large max_examples setting. This also works pretty well, but doesn't get the benefits of coverage guidance (i.e. avoiding the exponential scaling cliff by learning from feedback) and also occupies one fuzz target per core.

(turns out that you can emulate coverage guidance with hypothesis.target(), which appears to work well enough as a starting point)

(also Hypothesis used to have coverage guidance built in, but we took it back out because of performance and ecosystem integration problems - as a rule of thumb it's just not worth the trouble for less than a thousand inputs. see here.)

Features

• Interleave execution of many test functions
• Prioritise functions where we expect to make progress
• Coverage-guided exploration of your system-under-test
• Seamless python-native and CLI integrations

(see personal notes for more plans, and papers I'm cribbing from)

Changelog

Patch notes can be found in CHANGELOG.md.

License

This is an active research project as part of my (Zac Hatfield-Dodds) PhD.

Unlike Hypothesis, it is not open source and I am not seeking external contributions.

As a complement to users of free, world-class PBT tools, I'm planning to sell licenses in order to fund ongoing work on both this project and Hypothesis itself. Please contact me if you are interested.