Skip to content

Commit 5c3f66a

Browse files
author
Jamie Bloxham
authored
Change default DDP sync strategy to MULTI_AUTO_SYNC when find_unusued_parameters is set (#1087)
`FORCED_SYNC` currently seems to be unreliable. Switch to using `MULTI_AUTO_SYNC` instead when `find_unusued_parameters` is set. See #1086. Closes #492
1 parent f1862b8 commit 5c3f66a

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

composer/trainer/trainer.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ def _distribute_and_get_random_seed(seed: Optional[int], device: Device):
195195
def _get_ddp_sync_strategy(ddp_sync_strategy: Optional[Union[str, DDPSyncStrategy]], find_unused_parameters: bool):
196196
if ddp_sync_strategy is None:
197197
if find_unused_parameters:
198-
ddp_sync_strategy = DDPSyncStrategy.FORCED_SYNC
198+
ddp_sync_strategy = DDPSyncStrategy.MULTI_AUTO_SYNC
199199
else:
200200
ddp_sync_strategy = DDPSyncStrategy.SINGLE_AUTO_SYNC
201201
else:

0 commit comments

Comments
 (0)