tensorboard-reducer 0.1.0

Compute reduced statistics (mean, std, min, max, median or any other numpy operation) of multiple TensorBoard runs matching a directory glob pattern. This can be used after training multiple identical models to reduce the noise in their loss/accuracy/error curves e.g. when trying to establish a statistically significant improvement in training performance.

Requires PyTorch and TensorBoard. No TensorFlow installation required.

Installation

pip install tensorboard_reducer

Usage

The tb-reducer CLI has the following flags:

• -i/--indirs-glob (required): Glob pattern of the run directories to reduce.
• -o/--outdir (required): Name of the directory to save the new reduced run data.
• -r/--reduce-ops (required): Comma-separated names of numpy reduction ops (mean, std, min, max, ...). Default is mean. Each reduction is written to a separate outdir suffixed by its op name, e.g. if outdir='my-new-run, the mean reduction will be written to my-new-run-mean. Only exception is std which will create two outdirs named my-new-run-mean+std and my-new-run-mean-std.
• -f/--format: Output format of reduced TensorBoard runs. One of tb-events for regular TensorBoard event files or csv. If csv, -o/--outdir must have .csv extension and all reduction ops will be written to a single CSV file rather than separate directories for each reduce op.
• -w/--overwrite (optional): Whether to overwrite existing reduction directories.

Example:

tb-reducer -i 'glob_pattern/of_dirs_to_reduce*' -o basename_of_output_dir -r mean,std,min,max

Testing

This project uses pytest. To run the entire test suite:

python -m pytest

To run individual or groups of test files, pass pytest a path or glob pattern, respectively:

python -m pytest tests/test_cumulative.py
python -m pytest **/test_*_metrics.py

To run a single test, pass its name to the -k flag:

python -m pytest -k test_precision_recall_curve

Consult the pytest docs for more details.