A lightweight library for adding fault tolerance to large-scale PyTorch distributed training workloads.
Project description
torchsnapshot
This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.
A light-weight library for adding fault tolerance to large-scale PyTorch distributed training workloads.
Install
Requires Python >= 3.7 and PyTorch >= 1.11
From pip:
pip install --pre torchsnapshot-nightly
From source:
git clone https://github.com/facebookresearch/torchsnapshot
cd torchsnapshot
pip install -r requirements.txt
python setup.py install
Concepts
- Stateful object - an object that whose state can be obtained via
.state_dict()and restored via.load_state_dict(). Most PyTorch components (e.g.Module,Optimizer,LRScheduler) already implement this protocol. - App state - the application state described using multiple stateful objects.
- Snapshot - the persisted app state.
Basic Usage
Describing the application state with multiple stateful objects:
app_state = {"model": model, "optimizer": optimizer}
Taking a snapshot of the application state:
from torchsnapshot import Snapshot
# File System
snapshot = Snapshot.take(path="/foo/bar/baz", app_state=app_state)
# S3
snapshot = Snapshot.take(path="s3://foo/bar", app_state=app_state)
# Google Cloud Storage
snapshot = Snapshot.take(path="gcs://foo/bar", app_state=app_state)
Referencing an existing snapshot:
snapshot = Snapshot(path="foo/bar/baz")
Restoring the application state from a snapshot:
snapshot.restore(app_state=app_state)
See the example directory for more examples.
License
torchsnapshot is BSD licensed, as found in the LICENSE file.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file torchsnapshot-nightly-2022.7.7.tar.gz.
File metadata
- Download URL: torchsnapshot-nightly-2022.7.7.tar.gz
- Upload date:
- Size: 28.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.7.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b1365e13e97a917e90a33cc8b86998f632a0d0bee6bdbd6470197bc494c3880c |
|
| MD5 | b675a56508d5c85fdeaf982acc2ca44e |
|
| BLAKE2b-256 | c6e23a124e342a09092e5b7fd4035ab1f347caee7e5f5ed9fc7097c1748c5d57 |
Provenance
File details
Details for the file torchsnapshot_nightly-2022.7.7-py3-none-any.whl.
File metadata
- Download URL: torchsnapshot_nightly-2022.7.7-py3-none-any.whl
- Upload date:
- Size: 38.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.7.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 693448bb4dfaee012923aa6e9cafb5e9b5989c264c2d8a532f93b17dfb6a6df9 |
|
| MD5 | 3c885a58d0db4187c20aa641993c7197 |
|
| BLAKE2b-256 | a9fefdde0f083b62a162550c71ec189b9c1b6fd95195aa272c9bff35a9d2ba89 |
