Load callback states while testing.

[rohitgr7]

🚀 Feature

Load callback states while testing.

Motivation

#5161 (comment)

Pitch

Two possible API changes:

with an additional argument restore_states:

test(ckpt_path, restore_states=True/False)  # give an option whether to load states or not
test(model, ckpt_path, restore_states=True/False)  # same as above but will just load checkpoint states and not the model

# raise an error
test(ckpt_path=None, restore_states=True)

or without any additional argument:

test(ckpt_path)  # always load states
test(ckpt_path=None)  # don't load any states.
test(model, ckpt_path)  # reload checkpoint states only from ckpt_path

Alternatives

Alternatively, one can just reload checkpoints manually, call on_load_checkpoint for all the callbacks manually, and test.

PS: There may be a better solution. Open to suggestions :)
cc: @ananthsub

cc @Borda @awaelchli @ananthsub @ninginthecloud @rohitgr7 @tchaton @akihironitta

[awaelchli]

@rohitgr7 what's the difference between your proposal and the existing

trainer = Trainer(resume_from_checkpoint=x)
trainer.test()

?

[rohitgr7]

trainer = Trainer(resume_from_checkpoint=x)
trainer.test()

in the recent PR the reload from resume_from_checkpoint was disabled completely since it was reloading the model state too also resume_from_checkpoint is meant to resume the training I guess. If we use this then while testing with different checkpoints we need to pass that checkpoint path in both Trainer(resume_from_checkpoint=ckpt) and trainer.test(ckpt_path=ckpt). I think with resume_from_checkpoint it will be easier to handle, it's just that this argument name sounds a bit misleading to me in case of testing.

[ananthsub]

in the recent PR the reload from resume_from_checkpoint was disabled completely since it was reloading the model state too

Isn't this a breaking API change? I commented on #5388 about this too. What happens to callbacks whose states also depend on the model?

[ananthsub]

@rohitgr7 could we still call the checkpoint connector from setup_training inside run_evaluation, but add logic inside of restore to check for trainer.testing - if it's testing, we can load the model and callback states only and ignore the trainer states. this way we don't have to expose anything new in the trainer API. what do you think?

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[rohitgr7]

for this, I'd suggest reloading the callback states as well along with the model state using the ckpt_path passed in .test. Open to suggestions!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!