|
249 | 249 | "\n",
|
250 | 250 | "* [Logger][Logger]: Composer includes a centralized logger, which passes logged data to each `LoggerDestination` (more on that below). Logged data can be either metrics or artifacts. The logger is similar to Python's built-in `logging.getLogger(...)` but is designed to log structured metrics and artifacts in addition to just text.\n",
|
251 | 251 | "\n",
|
252 |
| - "[Logger]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.logger.html#composer.loggers.logger.Logger\n", |
| 252 | + "[Logger]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.Logger.html#logger\n", |
253 | 253 | "\n",
|
254 | 254 | "* [LoggerDestination][LoggerDestination]: Where logs are sent is specified via the `loggers` argument of the [Trainer constructor][trainer]. The centralized `Logger` (above) passes all metrics and artifacts to each `LoggerDestination`, which is responsible for handling and storing the data. For example, Composer includes `LoggerDestination`s for logging to [files][FileLogger], [TensorBoard][TensorboardLogger], [Weights & Biases][wandb], and [Object Stores][ObjectStoreLogger] like S3. Not all `LoggerDestination`s support storing all types of data; for example, the [ObjectStoreLogger][ObjectStoreLogger] only supports logging artifacts, whereas the [FileLogger][FileLogger] only supports logging files. Others, such as the [WandBLogger][wandb], support both.\n",
|
255 | 255 | "\n",
|
256 |
| - "[LoggerDestination]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.logger_destination.html#composer.loggers.logger_destination.LoggerDestination\n", |
257 |
| - "[trainer]: https://docs.mosaicml.com/en/stable/api_reference/composer.trainer.trainer.html#composer.trainer.trainer.Trainer\n", |
258 |
| - "[FileLogger]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.file_logger.html#composer.loggers.file_logger.FileLogger\n", |
259 |
| - "[TensorboardLogger]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.tensorboard_logger.html#composer.loggers.tensorboard_logger.TensorboardLogger\n", |
260 |
| - "[wandb]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.wandb_logger.html#composer.loggers.wandb_logger.WandBLogger\n", |
261 |
| - "[ObjectStoreLogger]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.object_store_logger.html#composer.loggers.object_store_logger.ObjectStoreLogger\n", |
262 |
| - "[FileLogger]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.file_logger.html#composer.loggers.file_logger.FileLogger\n", |
| 256 | + "[LoggerDestination]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.LoggerDestination.html\n", |
| 257 | + "[trainer]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.Trainer.html#trainer\n", |
| 258 | + "[FileLogger]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.FileLogger.html\n", |
| 259 | + "[TensorboardLogger]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.TensorboardLogger.html#tensorboardlogger\n", |
| 260 | + "[wandb]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.WandBLogger.html\n", |
| 261 | + "[ObjectStoreLogger]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.ObjectStoreLogger.html#objectstorelogger\n", |
| 262 | + "[FileLogger]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.loggers.FileLogger.html#filelogger\n", |
263 | 263 | "\n",
|
264 | 264 | "* Metrics: A metric is a scalar, such as accuracy, that can be logged. Usually you would want to plot metrics over time (e.g., to see how accuracy improves over batches).\n",
|
265 | 265 | "\n",
|
|
268 | 268 | "* [ObjectStore][ObjectStore]: The abstract `ObjectStore` class provides an API for uploading and downloading checkpoints. Composer includes object store implementations for [S3][S3], [SFTP][SFTP], and [Libcloud][Libcloud]. You can also write your own implementation by extending the [base class][ObjectStore] if you are using a custom backend.\n",
|
269 | 269 | "\n",
|
270 | 270 | "[Artifacts]: https://docs.mosaicml.com/en/stable/trainer/artifact_logging.html\n",
|
271 |
| - "[ObjectStore]: https://docs.mosaicml.com/en/stable/api_reference/composer.utils.object_store.object_store.html#composer.utils.object_store.object_store.ObjectStore\n", |
272 |
| - "[S3]: https://docs.mosaicml.com/en/stable/api_reference/composer.utils.object_store.s3_object_store.html#composer.utils.object_store.s3_object_store.S3ObjectStore\n", |
273 |
| - "[SFTP]: https://docs.mosaicml.com/en/stable/api_reference/composer.utils.object_store.sftp_object_store.html#composer.utils.object_store.sftp_object_store.SFTPObjectStore\n", |
274 |
| - "[Libcloud]: https://docs.mosaicml.com/en/stable/api_reference/composer.utils.object_store.libcloud_object_store.html#composer.utils.object_store.libcloud_object_store.LibcloudObjectStore\n", |
| 271 | + "[ObjectStore]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.utils.ObjectStore.html\n", |
| 272 | + "[S3]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.utils.S3ObjectStore.html\n", |
| 273 | + "[SFTP]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.utils.SFTPObjectStore.html\n", |
| 274 | + "[Libcloud]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.utils.LibcloudObjectStore.html\n", |
275 | 275 | "\n",
|
276 | 276 | "We'll use these components together to back up our checkpoints and TensorBoard TF Event files to the cloud. Internally, the [CheckpointSaver][CheckpointSaver] callback and [TensorboardLogger][TensorboardLogger] pass all generated files to the [Logger][Logger] as artifacts by calling [Logger.file_artifact][Logger_file_artifact]. The Logger then passes these files to each [LoggerDestination][LoggerDestination] (specified via the `loggers` argument of the [Trainer constructor][trainer]). A `LoggerDestination`, which can implement the `log_file_artifact` method, is responsible for uploading the file to the cloud.\n",
|
277 | 277 | "\n",
|
278 |
| - "[CheckpointSaver]: https://docs.mosaicml.com/en/stable/api_reference/composer.callbacks.checkpoint_saver.html#composer.callbacks.checkpoint_saver.CheckpointSaver\n", |
279 |
| - "[Logger_file_artifact]: https://docs.mosaicml.com/en/stable/api_reference/composer.loggers.logger.html#composer.loggers.logger.Logger.file_artifact\n", |
| 278 | + "[CheckpointSaver]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.callbacks.CheckpointSaver.html#checkpointsaver\n", |
| 279 | + "[Logger_file_artifact]: https://docs.mosaicml.com/en/stable/api_reference/generated/composer.Logger.html#composer.Logger.file_artifact\n", |
280 | 280 | "\n",
|
281 | 281 | "\n",
|
282 | 282 | "Here, our \"cloud\" will be an S3 bucket. To upload checkpoints and TensorBoard TF Event files to the bucket, we'll add the [ObjectStoreLogger][ObjectStoreLogger] with the [S3ObjectStore][S3] backend to our list of logger destinations. This class asynchronously uploads artifacts to an object store without blocking the training loop."
|
|
0 commit comments