Skip to main content

A lightweight library for adding fault tolerance to large-scale PyTorch distributed training workloads.

Project description

torchsnapshot

build status pypi version pypi nightly version codecov bsd license

This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.

A light-weight library for adding fault tolerance to large-scale PyTorch distributed training workloads.

Install

Requires Python >= 3.7 and PyTorch >= 1.11

From pip:

pip install --pre torchsnapshot-nightly

From source:

git clone https://github.com/facebookresearch/torchsnapshot
cd torchsnapshot
pip install -r requirements.txt
python setup.py install

Concepts

  • Stateful object - an object that whose state can be obtained via .state_dict() and restored via .load_state_dict(). Most PyTorch components (e.g. Module, Optimizer, LRScheduler) already implement this protocol.
  • App state - the application state described using multiple stateful objects.
  • Snapshot - the persisted app state.

Basic Usage

Describing the application state with multiple stateful objects:

app_state = {"model": model, "optimizer": optimizer}

Taking a snapshot of the application state:

from torchsnapshot import Snapshot

# File System
snapshot = Snapshot.take(path="/foo/bar/baz", app_state=app_state)

# S3
snapshot = Snapshot.take(path="s3://foo/bar", app_state=app_state)

# Google Cloud Storage
snapshot = Snapshot.take(path="gcs://foo/bar", app_state=app_state)

Referencing an existing snapshot:

snapshot = Snapshot(path="foo/bar/baz")

Restoring the application state from a snapshot:

snapshot.restore(app_state=app_state)

See the example directory for more examples.

License

torchsnapshot is BSD licensed, as found in the LICENSE file.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

torchsnapshot-nightly-2022.8.21.tar.gz (38.8 kB view details)

Uploaded Source

Built Distribution

torchsnapshot_nightly-2022.8.21-py3-none-any.whl (49.4 kB view details)

Uploaded Python 3

File details

Details for the file torchsnapshot-nightly-2022.8.21.tar.gz.

File metadata

File hashes

Hashes for torchsnapshot-nightly-2022.8.21.tar.gz
Algorithm Hash digest
SHA256 956a745db91c9ef1c532432c859cdc37462e0a110cd9ea62798e97795be55409
MD5 d5de86e3b012af73f8a1498c44801fb7
BLAKE2b-256 fb9e7b2e3451dbe27215ab65c1cc162ed669cca7fc992e9a47106e6ab70ce85d

See more details on using hashes here.

Provenance

File details

Details for the file torchsnapshot_nightly-2022.8.21-py3-none-any.whl.

File metadata

File hashes

Hashes for torchsnapshot_nightly-2022.8.21-py3-none-any.whl
Algorithm Hash digest
SHA256 67bac29232af2407920c750c7220cdc60b663fd4499bec525fec07ab06f854cb
MD5 b00eec91da965569f42e56ca790f70ef
BLAKE2b-256 57e10e7ee59cf4bac7f33fc432ad1ab2c81c06b898babbc1d4deac75bb159ffc

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page