A library for providing a simple interface to create new metrics and an easy-to-use toolkit for metric computations and checkpointing.
Project description
TorchEval
This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.
A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.
Installing TorchEval
Requires Python >= 3.7 and PyTorch >= 1.11
From pip:
pip install torcheval
For nighly build version
pip install --pre torcheval-nightly
From source:
git clone https://github.com/pytorch-labs/torcheval
cd torcheval
pip install -r requirements.txt
python setup.py install
Quick Start
cd torcheval
python examples/simple_example.py
Using TorchEval
TorchEval can be run on CPU, GPU, and Multi-GPUs/Muti-Nodes.
For the multiple devices usage:
import torch
from torcheval.metrics.toolkit import sync_and_compute
from torcheval.metrics import MulticlassAccuracy
local_rank = int(os.environ["LOCAL_RANK"])
global_rank = int(os.environ["RANK"])
world_size = int(os.environ["WORLD_SIZE"])
device = torch.device(
f"cuda:{local_rank}"
if torch.cuda.is_available() and torch.cuda.device_count() >= world_size
else "cpu"
)
metric = MulticlassAccuracy(device=device)
num_epochs, num_batches = 4, 8
for epoch in range(num_epochs):
for i in range(num_batches):
input = torch.randint(high=5, size=(10,), device=device)
target = torch.randint(high=5, size=(10,), device=device)
# metric.update() updates the metric state with new data
metric.update(preds, target)
# metric.compute() returns metric value from all seen data on the local process.
local_compute_result = metric.compute()
# sync_and_compute(metric) returns metric value from all seen data on all processes.
# It gives the same result as ``metric.compute()`` if it's run on single process.
global_compute_result = sync_and_compute(metric)
# The final result is collected by rank 0
if global_rank == 0:
print(global_compute_result)
# metric.reset() cleans up all seen data
metric.reset()
See the example directory for more examples.
Contributing
We welcome PRs! See the CONTRIBUTING file.
License
TorchEval is BSD licensed, as found in the LICENSE file.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for torcheval-nightly-2022.8.13.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fac6f9fe40b4d53b5a238db0170b5bb43b2a5ded89dc8faca19a94d89360cdb5 |
|
| MD5 | 0780d8ef3f7ab48b378d8c29807e62d3 |
|
| BLAKE2b-256 | c9280b3bd89e84b1b06a0d20084d49813130f2f4e1f78092b3494c4541791d33 |
Hashes for torcheval_nightly-2022.8.13-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 13ca8bbd7ba69cfcbe8f02a36fa72200a6665442e9fed08c8ee0fb8510ead01a |
|
| MD5 | 5030b15eb66e6eef96fcf4445905d8b4 |
|
| BLAKE2b-256 | a92cec6a876258d11f519f504ad60f2b6fab0f0b1db79de1219f35f78cf30f59 |
