A performant, memory-efficient checkpointing library for PyTorch applications, designed with large, complex distributed workloads in mind.
Project description
torchsnapshot
This library is currently in Alpha and currently does not have a stable release. The API may change and may not be backward compatible. If you have suggestions for improvements, please open a GitHub issue. We'd love to hear your feedback.
A performant, memory-efficient checkpointing library for PyTorch applications, designed with large, complex distributed workloads in mind.
Install
Requires Python >= 3.7 and PyTorch >= 1.12
From pip:
pip install --pre torchsnapshot-nightly
From source:
git clone https://github.com/pytorch/torchsnapshot
cd torchsnapshot
pip install -r requirements.txt
python setup.py install
Why TorchSnapshot
Performance
- TorchSnapshot provides a fast checkpointing implementation employing various optimizations, including zero-copy serialization for most tensor types, overlapped device-to-host copy and storage I/O, parallelized storage I/O.
- TorchSnapshot greatly speeds up checkpointing for DistributedDataParallel workloads by distributing the write load across all ranks (benchmark).
- When host memory is abundant, TorchSnapshot allows training to resume before all storage I/O completes, reducing the time blocked by checkpoint saving.
Memory Usage
- TorchSnapshot's memory usage adapts to the host's available resources, greatly reducing the chance of out-of-memory issues when saving and loading checkpoints.
- TorchSnapshot supports efficient random access to individual objects within a snapshot, even when the snapshot is stored in a cloud object storage.
Usability
- Simple APIs that are consistent between distributed and non-distributed workloads.
- Out of the box integration with commonly used cloud object storage systems.
- Automatic resharding (elasticity) on world size change for supported workloads (more details).
Security
- Secure tensor serialization without pickle dependency [WIP].
Getting Started
from torchsnapshot import Snapshot
# Taking a snapshot
app_state = {"model": model, "optimizer": optimizer}
snapshot = Snapshot.take(app_state=app_state, "/path/to/snapshot")
# Restoring from a snapshot
snapshot.restore(app_state=app_state)
See the documentation for more details.
License
torchsnapshot is BSD licensed, as found in the LICENSE file.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file torchsnapshot-nightly-2022.10.17.tar.gz.
File metadata
- Download URL: torchsnapshot-nightly-2022.10.17.tar.gz
- Upload date:
- Size: 48.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.7.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1f122da2ce5b4c28ecab4fbbb165d7ccd765be20b25ac8680c872c185deeb935 |
|
| MD5 | 7f0c64852531b6ca4c5b28eb03faeff1 |
|
| BLAKE2b-256 | 427eb761879c3e68321bb233f6d71c14ef3705193b1f8df01b944bfa75af0c1f |
Provenance
File details
Details for the file torchsnapshot_nightly-2022.10.17-py3-none-any.whl.
File metadata
- Download URL: torchsnapshot_nightly-2022.10.17-py3-none-any.whl
- Upload date:
- Size: 59.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.7.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 60eaf3f1f6131db407b5e03cce06af225d5909094d3a2509014e84fa281554ab |
|
| MD5 | 232476e8c10a1e3a77be26638c51588a |
|
| BLAKE2b-256 | 34c9e64a0867c880e461683d630e8d4168714a245f488f4d21ace609f2875d89 |
