Load checkpoints from cloud storage #200
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added support to load checkpoints stored in object storage (rather than just on the local disk). Closes #192. - Refactored the run directory uploader to separate out object store related utilites to composer.utils.object_store (and added test coverage). - Updated the checkpointer hparams to optionally take `composer.utils.object_store.ObjectStoreProviderHparams`, which would be used to download the checkpoint from storage. - Updated the trainer init to propegate through this change.
|
Going to re-open this PR once #199 is merged in and this PR is rebased on top of it. |
Refactored checkpoint storage
|
I've tested this and it works! Great job Ravi! |
moinnadeem
approved these changes
Jan 7, 2022
coryMosaicML
pushed a commit
to coryMosaicML/composer
that referenced
this pull request
Feb 23, 2022
Added support to load checkpoints stored in object storage (rather than just on the local disk) and from URLs. Closes mosaicml#192. - Refactored the run directory uploader to separate out object store related utilities to `composer.utils.object_store` (and added test coverage). - Updated the checkpointer loader and hparams to support checkpoints stored in URLs and in object stores. Chunk sizing and whether to show tqdm bars are hparams. - Refactored how ddp checkpoints are loaded and stored to avoid an `os.listdir` and writing temporary files to the run directory - Updated the `Trainer.__init__` to propagate through this change.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added support to load checkpoints stored in object storage (rather than just on the local disk) and from URLs. Closes #192.
composer.utils.object_store(and added test coverage).os.listdirand writing temporary files to the run directoryTrainer.__init__to propagate through this change.