Skip to main content

Richbench, a little benchmarking tool

Project description

rich-bench

A little Python benchmarking tool.

Why do I need this?

The builtin timeit module for Python is great, but the typical usage for micro-benchmarks is to run a small script like this:

python -m timeit "a = 1; b = 2; a * b"

The problem with this approach is that the compiled code is a module, so any variables on the top-level are globals. The compiled bytecode is different to the same statements being inside a local scope (e.g. a method or function). They behave and perform very differently in CPython.

richbench encourages you to write benchmarks inside functions to properly simulate the closures and scope of production code.

Installation

Requires Python 3.6+ and can be installed using pip:

pip install richbench

Usage

Write your benchmark functions in a directory and run richbench with that target to get the results:

$ richbench my_benchmarks/

Results are displayed in a table like this:

Example result table

$ richbench --help
usage: richbench [-h] [--profile] [--percentage] [--markdown] [--benchmark [BENCHMARK]] [--repeat REPEAT] [--times TIMES] target [target ...]

positional arguments:
  target

options:
  -h, --help            show this help message and exit
  --profile             Profile the benchmarks and store in .profiles/
  --percentage          Show percentage of improvement instead of multiplier
  --markdown            Prints a markdown friendly table
  --benchmark [BENCHMARK]
                        Run specific benchmark
  --repeat REPEAT       Repeat benchmark this many times
  --times TIMES         Run benchmark this many times

Writing benchmarks

Benchmarks should be in a directory and must have the filename bench_{name}.py.

The last statement in the benchmark file should be a list, called __benchmarks__ with a list of tuples containing:

  1. function a
  2. function b
  3. the name of the benchmark
def sort_seven():
    """Sort a list of seven items"""
    for _ in range(10_000):
        sorted([3,2,4,5,1,5,3])

def sort_three():
    """Sort a list of three items"""
    for _ in range(10_000):
        sorted([3,2,4])

__benchmarks__ = [
    (sort_seven, sort_three, "Sorting 3 items instead of 7")
]

Tips for benchmarking

Inside your benchmark function try to:

  • Run setup commands once
  • Repeat the code you want to benchmark as many times as possible to create a stable benchmark

For micro-benchmarks, this can be achieved by calling 1000 - 100,000 times.

If your benchmark code completes within 0.001s it is likely you will get unstable benchmarks because of the CPU doing other activities. Increase the times the target code is run within the function.

Profiling

By adding the --profile flag to the command line, it will generate a subdirectory .profiles with HTML profile data of your target functions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

richbench-1.0.3.tar.gz (1.1 MB view details)

Uploaded Source

Built Distribution

richbench-1.0.3-py3-none-any.whl (5.2 kB view details)

Uploaded Python 3

File details

Details for the file richbench-1.0.3.tar.gz.

File metadata

  • Download URL: richbench-1.0.3.tar.gz
  • Upload date:
  • Size: 1.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-requests/2.26.0

File hashes

Hashes for richbench-1.0.3.tar.gz
Algorithm Hash digest
SHA256 744afa3e78cbd919721042c11f7b7f9d2f546cebb3333d40290c4a0d88791701
MD5 7fecc5eaff5e15fdad344fabba1e6d6b
BLAKE2b-256 9e5bd400ecddd716348ff7eac2a40df67cde1c9b60b1f7083577605db4bd4b31

See more details on using hashes here.

File details

Details for the file richbench-1.0.3-py3-none-any.whl.

File metadata

  • Download URL: richbench-1.0.3-py3-none-any.whl
  • Upload date:
  • Size: 5.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-requests/2.26.0

File hashes

Hashes for richbench-1.0.3-py3-none-any.whl
Algorithm Hash digest
SHA256 f52651cc0e0069a1355c5ed8cda214cb3f8961e7aaa431e440071d30f62e3e55
MD5 46c2ab4bbe33c060a6e4563445b8e051
BLAKE2b-256 caa2725f87821d93f82cba818037fba0b86b14c721d75d4e9935b441ced8471f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page