Skip to main content

Python module to run and analyze benchmarks

Project description

Latest release on the Python Cheeseshop (PyPI) Build status of pyperf on GitHub Actions

The Python pyperf module is a toolkit to write, run and analyze benchmarks.

Features

  • Simple API to run reliable benchmarks

  • Automatically calibrate a benchmark for a time budget.

  • Spawn multiple worker processes.

  • Compute the mean and standard deviation.

  • Detect if a benchmark result seems unstable.

  • JSON format to store benchmark results.

  • Support multiple units: seconds, bytes and integer.

Usage

To run a benchmark use the pyperf timeit command (result written into bench.json):

$ python3 -m pyperf timeit '[1,2]*1000' -o bench.json
.....................
Mean +- std dev: 4.22 us +- 0.08 us

Or write a benchmark script bench.py:

#!/usr/bin/env python3
import pyperf

runner = pyperf.Runner()
runner.timeit(name="sort a sorted list",
              stmt="sorted(s, key=f)",
              setup="f = lambda x: x; s = list(range(1000))")

See the API docs for full details on the timeit function and the Runner class. To run the script and dump the results into a file named bench.json:

$ python3 bench.py -o bench.json

To analyze benchmark results use the pyperf stats command:

$ python3 -m pyperf stats telco.json
Total duration: 29.2 sec
Start date: 2016-10-21 03:14:19
End date: 2016-10-21 03:14:53
Raw value minimum: 177 ms
Raw value maximum: 183 ms

Number of calibration run: 1
Number of run with values: 40
Total number of run: 41

Number of warmup per run: 1
Number of value per run: 3
Loop iterations per value: 8
Total number of values: 120

Minimum:         22.1 ms
Median +- MAD:   22.5 ms +- 0.1 ms
Mean +- std dev: 22.5 ms +- 0.2 ms
Maximum:         22.9 ms

  0th percentile: 22.1 ms (-2% of the mean) -- minimum
  5th percentile: 22.3 ms (-1% of the mean)
 25th percentile: 22.4 ms (-1% of the mean) -- Q1
 50th percentile: 22.5 ms (-0% of the mean) -- median
 75th percentile: 22.7 ms (+1% of the mean) -- Q3
 95th percentile: 22.9 ms (+2% of the mean)
100th percentile: 22.9 ms (+2% of the mean) -- maximum

Number of outlier (out of 22.0 ms..23.0 ms): 0

There’s also:

  • pyperf compare_to command tests if a difference is significant. It supports comparison between multiple benchmark suites (made of multiple benchmarks)

    $ python3 -m pyperf compare_to --table mult_list_py36.json mult_list_py37.json mult_list_py38.json
    +----------------+----------------+-----------------------+-----------------------+
    | Benchmark      | mult_list_py36 | mult_list_py37        | mult_list_py38        |
    +================+================+=======================+=======================+
    | [1]*1000       | 2.13 us        | 2.09 us: 1.02x faster | not significant       |
    +----------------+----------------+-----------------------+-----------------------+
    | [1,2]*1000     | 3.70 us        | 5.28 us: 1.42x slower | 3.18 us: 1.16x faster |
    +----------------+----------------+-----------------------+-----------------------+
    | [1,2,3]*1000   | 4.61 us        | 6.05 us: 1.31x slower | 4.17 us: 1.11x faster |
    +----------------+----------------+-----------------------+-----------------------+
    | Geometric mean | (ref)          | 1.22x slower          | 1.09x faster          |
    +----------------+----------------+-----------------------+-----------------------+
  • pyperf system tune command to tune your system to run stable benchmarks.

  • Automatically collect metadata on the computer and the benchmark: use the pyperf metadata command to display them, or the pyperf collect_metadata command to manually collect them.

  • --track-memory and --tracemalloc options to track the memory usage of a benchmark.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyperf-2.8.0.tar.gz (225.2 kB view details)

Uploaded Source

Built Distribution

pyperf-2.8.0-py3-none-any.whl (142.5 kB view details)

Uploaded Python 3

File details

Details for the file pyperf-2.8.0.tar.gz.

File metadata

  • Download URL: pyperf-2.8.0.tar.gz
  • Upload date:
  • Size: 225.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.6

File hashes

Hashes for pyperf-2.8.0.tar.gz
Algorithm Hash digest
SHA256 b30a20465819daf102b6543b512f6799a5a879ff2a123981e6cd732d0e6a7a79
MD5 aac40171c00dd88742ffc2f339a7dc6a
BLAKE2b-256 022a758b3c4cc9843bd385bc595b777345fbf4cd00733b7830cdff43e30002c0

See more details on using hashes here.

Provenance

File details

Details for the file pyperf-2.8.0-py3-none-any.whl.

File metadata

  • Download URL: pyperf-2.8.0-py3-none-any.whl
  • Upload date:
  • Size: 142.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.6

File hashes

Hashes for pyperf-2.8.0-py3-none-any.whl
Algorithm Hash digest
SHA256 1a775b5a09882f18bf876430ef78e07646f773f50774546f5f6a8b34d60e3968
MD5 ec04cfbceb74690a502cd66521a4149e
BLAKE2b-256 7ff7bb8965520a9b0a3d720b282e67b5cb7f3305b96e4bacaee2794550e67e94

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page