I can't resume self-play training

[efimmaksimov]

When I try to resume self-play training using mlagents-learn <trainer-config-file> --run-id=<run-identifier> --resume I get an error:

2020-10-21 19:00:59 INFO [tf_model_saver.py:108] Loading model from results\Tenn
is1\Tennis.
2020-10-21 19:01:00 INFO [model_serialization.py:205] List of nodes to export fo
r behavior :Tennis
2020-10-21 19:01:00 INFO [model_serialization.py:207]   is_continuous_control
2020-10-21 19:01:00 INFO [model_serialization.py:207]   trainer_major_version
2020-10-21 19:01:00 INFO [model_serialization.py:207]   trainer_minor_version
2020-10-21 19:01:00 INFO [model_serialization.py:207]   trainer_patch_version
2020-10-21 19:01:00 INFO [model_serialization.py:207]   version_number
2020-10-21 19:01:00 INFO [model_serialization.py:207]   memory_size
2020-10-21 19:01:00 INFO [model_serialization.py:207]   action_output_shape
2020-10-21 19:01:00 INFO [model_serialization.py:207]   action
2020-10-21 19:01:00 INFO [model_serialization.py:207]   action_probs
Converting results\Tennis1\Tennis/frozen_graph_def.pb to results\Tennis1\Tennis\
Tennis-10788.nn
IGNORED: Cast unknown layer
IGNORED: Shape unknown layer
IGNORED: StopGradient unknown layer
GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_versio
n', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_sha
pe'
IN: 'vector_observation': [-1, 1, 1, 27] => 'sub_2'
OUT: 'action', 'action_probs'
DONE: wrote results\Tennis1\Tennis\Tennis-10788.nn file.
2020-10-21 19:01:01 INFO [model_serialization.py:87] Exported results\Tennis1\Te
nnis\Tennis-10788.nn
2020-10-21 19:01:01 INFO [tf_model_saver.py:166] Copied results\Tennis1\Tennis\T
ennis-10788.nn to results\Tennis1\Tennis.nn.
2020-10-21 19:01:01 INFO [trainer_controller.py:84] Saved Model
Traceback (most recent call last):
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\tens
orflow\python\client\session.py", line 1121, in _run
    subfeed, allow_tensor=True, allow_operation=False)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\tens
orflow\python\framework\ops.py", line 3670, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\tens
orflow\python\framework\ops.py", line 3712, in _as_graph_element_locked
    "graph." % (repr(name), repr(op_name)))
KeyError: "The name 'save/Const:0' refers to a Tensor which does not exist. The
operation, 'save/Const', does not exist in the graph."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\daniil\AppData\Local\Programs\Python\Python37\lib\runpy.py", li
ne 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\daniil\AppData\Local\Programs\Python\Python37\lib\runpy.py", li
ne 85, in _run_code
    exec(code, run_globals)
  File "C:\unity\projects\ML-agent\python-envs\sample-env\Scripts\mlagents-learn
.exe\__main__.py", line 7, in <module>
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\learn.py", line 278, in main
    run_cli(parse_command_line())
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\learn.py", line 274, in run_cli
    run_training(run_seed, options)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\learn.py", line 151, in run_training
    tc.start_learning(env_manager)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents_envs\timers.py", line 305, in wrapped
    return func(*args, **kwargs)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\trainer_controller.py", line 172, in start_learning
    self._reset_env(env_manager)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents_envs\timers.py", line 305, in wrapped
    return func(*args, **kwargs)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\trainer_controller.py", line 110, in _reset_env
    self._register_new_behaviors(env_manager, env_manager.first_step_infos)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\trainer_controller.py", line 266, in _register_new_behaviors
    self._create_trainers_and_managers(env_manager, new_behavior_ids)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\trainer_controller.py", line 164, in _create_trainers_and_managers
    self._create_trainer_and_manager(env_manager, behavior_id)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\trainer_controller.py", line 138, in _create_trainer_and_manager
    parsed_behavior_id, env_manager.training_behaviors[name_behavior_id]
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\ghost\trainer.py", line 322, in create_policy
    self.trainer.model_saver.initialize_or_load(policy)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\model_saver\tf_model_saver.py", line 97, in initialize_or_load
    self._load_graph(policy, self.model_path, reset_global_steps=reset_steps)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\mlag
ents\trainers\model_saver\tf_model_saver.py", line 119, in _load_graph
    self.tf_saver.restore(policy.sess, ckpt.model_checkpoint_path)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\tens
orflow\python\training\saver.py", line 1299, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\tens
orflow\python\client\session.py", line 958, in run
    run_metadata_ptr)
  File "c:\unity\projects\ml-agent\python-envs\sample-env\lib\site-packages\tens
orflow\python\client\session.py", line 1124, in _run
    e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: The name 'save/Const:0' ref
ers to a Tensor which does not exist. The operation, 'save/Const', does not exis
t in the graph.

This error only appears when I try to resume self-play training. I tried 3 environments using self-play: SoccerTwos, Tennis, and my own. I also tried to resume training in other environments: 3DBall, FootCollector, RollerBall, which I did in the tutorial. In these environments, there was no error and training resumed.

[chriselion]

Hi,
I can reproduce the error on the latest release (release_8). I'll look in this and get back to you soon on a fix.

Also, I made a minor edit to your comment to use ``` to make the logs clearer. Hope you don't mind.

[chriselion]

We're working on a fix in #4593. We'll probably do a patch release with this fix sometime next week (depending on whether any other issues show up).

[CubeMD]

Hi,
Both resume and initialize-from parameters result in the same issue in release 7 when using self-play.

[chriselion]

Thanks @CubeMD, we'll make sure the fix handles both cases.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 14 days. It will be closed in the next 14 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last 28 days. If this issue is still valid, please ping a maintainer. Thank you for your contributions.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.