Skip to main content

A library for providing a simple interface to create new metrics and an easy-to-use toolkit for metric computations and checkpointing.

Project description

TorchEval

build status pypi version pypi nightly version bsd license docs

This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.

A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.

Installing TorchEval

Requires Python >= 3.7 and PyTorch >= 1.11

From pip:

pip install torcheval

For nighly build version

pip install --pre torcheval-nightly

From source:

git clone https://github.com/pytorch-labs/torcheval
cd torcheval
pip install -r requirements.txt
python setup.py install

Quick Start

cd torcheval
python examples/simple_example.py

Documentation

Documentation can be found at at pytorch-labs.github.io/torcheval

Using TorchEval

TorchEval can be run on CPU, GPU, and Multi-GPUs/Muti-Nodes.

For the multiple devices usage:

import torch
from torcheval.metrics.toolkit import sync_and_compute
from torcheval.metrics import MulticlassAccuracy

local_rank = int(os.environ["LOCAL_RANK"])
global_rank = int(os.environ["RANK"])
world_size  = int(os.environ["WORLD_SIZE"])

device = torch.device(
    f"cuda:{local_rank}"
    if torch.cuda.is_available() and torch.cuda.device_count() >= world_size
    else "cpu"
)

metric = MulticlassAccuracy(device=device)
num_epochs, num_batches = 4, 8

for epoch in range(num_epochs):
    for i in range(num_batches):
        input = torch.randint(high=5, size=(10,), device=device)
        target = torch.randint(high=5, size=(10,), device=device)

        # metric.update() updates the metric state with new data
        metric.update(input, target)


        # metric.compute() returns metric value from all seen data on the local process.
        local_compute_result = metric.compute()

        # sync_and_compute(metric) returns metric value from all seen data on all processes.
        # It gives the same result as ``metric.compute()`` if it's run on single process.
        global_compute_result = sync_and_compute(metric)

        # The final result is collected by rank 0
        if global_rank == 0:
            print(global_compute_result)

    # metric.reset() cleans up all seen data
    metric.reset()

See the example directory for more examples.

Contributing

We welcome PRs! See the CONTRIBUTING file.

License

TorchEval is BSD licensed, as found in the LICENSE file.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

torcheval-nightly-2022.9.17.tar.gz (57.9 kB view details)

Uploaded Source

Built Distribution

torcheval_nightly-2022.9.17-py3-none-any.whl (104.3 kB view details)

Uploaded Python 3

File details

Details for the file torcheval-nightly-2022.9.17.tar.gz.

File metadata

  • Download URL: torcheval-nightly-2022.9.17.tar.gz
  • Upload date:
  • Size: 57.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.7.13

File hashes

Hashes for torcheval-nightly-2022.9.17.tar.gz
Algorithm Hash digest
SHA256 7bdf5492b7b0309a608256ace7a93d9a702a65a92fdf4bf9bc73d3fed0f8cc08
MD5 3b4777a7b55b3a7631561d676d4d5f1a
BLAKE2b-256 bdc9f728c9e6b792b46800330e65c6aa03989fd9c710b4541d5b54317adfa1c3

See more details on using hashes here.

Provenance

File details

Details for the file torcheval_nightly-2022.9.17-py3-none-any.whl.

File metadata

File hashes

Hashes for torcheval_nightly-2022.9.17-py3-none-any.whl
Algorithm Hash digest
SHA256 99c18a5e2a8379336a7de70527be39899b9e126867511fcd0e05b517200be269
MD5 9268360823d3f3825dc0e36196363bb8
BLAKE2b-256 f22eb93f13022a8b2f97ebbfb1dccb260c1c09f305cbc4f38e87a33710b904f6

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page