[refactor] Make PyTorch the default and TensorFlow optional (#4517)

* Torch setup.py

* Set torch to default

* Make torch default in setup.py

* Remove indents

* Remove other instances of TF being used

* Add tensorboard to setup.py

* Adding correst setup commands for verifying torch is installed (#4524)

* Adding correst setup commands for verifying torch is installed

* Editing the test_requirments to add tf and remove torch

* Develop torchdefault raise outside setup (#4530)

* Torch not imported error to raise at first usage

* Torch not imported error to raise at first usage

* [refactor] Use PyTorch TensorBoard utils (#4518)

* Convert stats writer to use PyTorch TB support

* Use common function to print params

* Update test

* Bump tensorboard to 1.15 to fix the tests

* putting tensorboard 1.15.0 as min version requirement

Co-authored-by: vincentpierre <[email protected]>

* [Docs] Initial documentation changes for making Torch the default (#4561)

* Initial commit

* Forgotten doc

* Removing the `Installation-Anaconda-Windows.md` as it is deprecated

* Readding the depreacted Installation-Anaconda-Windows.md but leaving it unchanged

* more references to tensorflow removed

* Update README.md

Co-authored-by: Ervin T. <[email protected]>

* Change references to .nn to .onnx in docs (#4583)

Co-authored-by: Ervin T. <[email protected]>

* [refactor] Add --tensorflow, enable Torch as default setting (#4582)

* Add --tensorflow option

* Switch framework to Pytorch default

* Update changelog

* Re-add --torch

* Edit warning

* Modify Yamato tests (#4584)

* Don't check for PB file in Yamato inference

* Only run inference on ONNX

* Update docs/Unity-Inference-Engine.md with correct tf2onnx versions

Co-authored-by: Chris Elion <[email protected]>

* Add reward signal class comments

* More descriptive import of is_available

* Updated installation instructions for PyTorch

* More reward signal comments

* More Windows instructions

Co-authored-by: Vincent-Pierre BERGES <[email protected]>
Co-authored-by: Chris Elion <[email protected]>

Author: 3 people

.github/ISSUE_TEMPLATE/bug_report.md

@@ -27,7 +27,7 @@ If applicable, add screenshots to help explain your problem.
 - Unity Version: [e.g. Unity 2020.1f1]
 - OS + version: [e.g. Windows 10]
 - _ML-Agents version_: (e.g. ML-Agents v0.8, or latest `develop` branch from source)
-- _TensorFlow version_: (you can run `pip3 show tensorflow` to get this)
+- _Torch version_: (you can run `pip3 show torch` to get this)
 - _Environment_: (which example environment you used to reproduce the error)
 
 **NOTE:** We are unable to help reproduce bugs with custom environments.  Please attempt to reproduce your issue with one of the example environments, or provide a minimal patch to one of the environments needed to reproduce the issue.

README.md

@@ -13,7 +13,7 @@
 project that enables games and simulations to serve as environments for
 training intelligent agents. Agents can be trained using reinforcement learning,
 imitation learning, neuroevolution, or other machine learning methods through a
-simple-to-use Python API. We also provide implementations (based on TensorFlow)
+simple-to-use Python API. We also provide implementations (based on PyTorch)
 of state-of-the-art algorithms to enable game developers and hobbyists to easily
 train intelligent agents for 2D, 3D and VR/AR games. These trained agents can be
 used for multiple purposes, including controlling NPC behavior (in a variety of

com.unity.ml-agents/CHANGELOG.md

@@ -12,6 +12,11 @@ and this project adheres to
 ### Major Changes
 #### com.unity.ml-agents (C#)
 #### ml-agents / ml-agents-envs / gym-unity (Python)
+ - PyTorch trainers are now the default. See the
+ [installation docs](https://github.com/Unity-Technologies/ml-agents/blob/mastere/docs/Installation.md) for
+ more information on installing PyTorch. For the time being, TensorFlow is still available;
+ you can use the TensorFlow backend by adding `--tensorflow` to the CLI, or
+ adding `framework: tensorflow` in the configuration YAML. (#4517)
 
 ### Minor Changes
 #### com.unity.ml-agents (C#)

docs/Background-Machine-Learning.md

@@ -194,4 +194,4 @@ we can learn policies for very complex environments (a complex environment is
 one where the number of observations an agent perceives and the number of
 actions they can take are large). Many of the algorithms we provide in ML-Agents
 use some form of deep learning, built on top of the open-source library,
-[TensorFlow](Background-TensorFlow.md).
+[PyTorch](Background-PyTorch.md).

docs/Background-TensorFlow.md renamed to docs/Background-PyTorch.md

@@ -1,29 +1,29 @@
-# Background: TensorFlow
+# Background: PyTorch
 
 As discussed in our
 [machine learning background page](Background-Machine-Learning.md), many of the
 algorithms we provide in the ML-Agents Toolkit leverage some form of deep
 learning. More specifically, our implementations are built on top of the
-open-source library [TensorFlow](https://www.tensorflow.org/). In this page we
-provide a brief overview of TensorFlow, in addition to TensorFlow-related tools
+open-source library [PyTorch](https://pytorch.org/). In this page we
+provide a brief overview of PyTorch and TensorBoard
 that we leverage within the ML-Agents Toolkit.
 
-## TensorFlow
+## PyTorch
 
-[TensorFlow](https://www.tensorflow.org/) is an open source library for
+[PyTorch](https://pytorch.org/) is an open source library for
 performing computations using data flow graphs, the underlying representation of
 deep learning models. It facilitates training and inference on CPUs and GPUs in
 a desktop, server, or mobile device. Within the ML-Agents Toolkit, when you
-train the behavior of an agent, the output is a model (.nn) file that you can
+train the behavior of an agent, the output is a model (.onnx) file that you can
 then associate with an Agent. Unless you implement a new algorithm, the use of
-TensorFlow is mostly abstracted away and behind the scenes.
+PyTorch is mostly abstracted away and behind the scenes.
 
 ## TensorBoard
 
-One component of training models with TensorFlow is setting the values of
+One component of training models with PyTorch is setting the values of
 certain model attributes (called _hyperparameters_). Finding the right values of
 these hyperparameters can require a few iterations. Consequently, we leverage a
-visualization tool within TensorFlow called
+visualization tool called
 [TensorBoard](https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard).
 It allows the visualization of certain agent attributes (e.g. reward) throughout
 training which can be helpful in both building intuitions for the different

docs/Getting-Started.md

@@ -91,7 +91,7 @@ itself to keep the ball balanced on its head.
 
 ## Running a pre-trained model
 
-We include pre-trained models for our agents (`.nn` files) and we use the
+We include pre-trained models for our agents (`.onnx` files) and we use the
 [Unity Inference Engine](Unity-Inference-Engine.md) to run these models inside
 Unity. In this section, we will use the pre-trained model for the 3D Ball
 example.
@@ -124,7 +124,7 @@ example.
 
 ## Training a new model with Reinforcement Learning
 
-While we provide pre-trained `.nn` files for the agents in this environment, any
+While we provide pre-trained models for the agents in this environment, any
 environment you make yourself will require training agents from scratch to
 generate a new model file. In this section we will demonstrate how to use the
 reinforcement learning algorithms that are part of the ML-Agents Python package
@@ -229,7 +229,7 @@ Once the training process completes, and the training process saves the model
 use it with compatible Agents (the Agents that generated the model). **Note:**
 Do not just close the Unity Window once the `Saved Model` message appears.
 Either wait for the training process to close the window or press `Ctrl+C` at
-the command-line prompt. If you close the window manually, the `.nn` file
+the command-line prompt. If you close the window manually, the `.onnx` file
 containing the trained model is not exported into the ml-agents folder.
 
 If you've quit the training early using `Ctrl+C` and want to resume training,
@@ -239,7 +239,7 @@ run the same command again, appending the `--resume` flag:
 mlagents-learn config/ppo/3DBall.yaml --run-id=first3DBallRun --resume
 ```
 
-Your trained model will be at `results/<run-identifier>/<behavior_name>.nn` where
+Your trained model will be at `results/<run-identifier>/<behavior_name>.onnx` where
 `<behavior_name>` is the name of the `Behavior Name` of the agents corresponding
 to the model. This file corresponds to your model's latest checkpoint. You can
 now embed this trained model into your Agents by following the steps below,
@@ -249,7 +249,7 @@ which is similar to the steps described [above](#running-a-pre-trained-model).
    `Project/Assets/ML-Agents/Examples/3DBall/TFModels/`.
 1. Open the Unity Editor, and select the **3DBall** scene as described above.
 1. Select the **3DBall** prefab Agent object.
-1. Drag the `<behavior_name>.nn` file from the Project window of the Editor to
+1. Drag the `<behavior_name>.onnx` file from the Project window of the Editor to
    the **Model** placeholder in the **Ball3DAgent** inspector window.
 1. Press the **Play** button at the top of the Editor.
 

docs/Installation.md

@@ -44,11 +44,6 @@ If your Python environment doesn't include `pip3`, see these
 [instructions](https://packaging.python.org/guides/installing-using-linux-tools/#installing-pip-setuptools-wheel-with-linux-package-managers)
 on installing it.
 
-Although we do not provide support for Anaconda installation on Windows, the
-previous
-[Windows Anaconda Installation (Deprecated) guide](Installation-Anaconda-Windows.md)
-is still available.
-
 ### Clone the ML-Agents Toolkit Repository (Optional)
 
 Now that you have installed Unity and Python, you can now install the Unity and
@@ -124,6 +119,22 @@ Virtual Environments. Virtual Environments provide a mechanism for isolating the
 dependencies for each project and are supported on Mac / Windows / Linux. We
 offer a dedicated [guide on Virtual Environments](Using-Virtual-Environment.md).
 
+#### (Windows) Installing PyTorch
+
+On Windows, you'll have to install the PyTorch package separately prior to
+installing ML-Agents. Activate your virtual environment and run from the command line:
+
+```sh
+pip3 install torch -f https://download.pytorch.org/whl/torch_stable.html
+```
+
+Note that on Windows, you may also need Microsoft's
+[Visual C++ Redistributable](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads)
+if you don't have it already. See the [PyTorch installation guide](https://pytorch.org/get-started/locally/)
+for more installation options and versions.
+
+#### Installing `mlagents`
+
 To install the `mlagents` Python package, activate your virtual environment and
 run from the command line:
 
@@ -138,7 +149,7 @@ line parameters you can use with `mlagents-learn`.
 
 By installing the `mlagents` package, the dependencies listed in the
 [setup.py file](../ml-agents/setup.py) are also installed. These include
-[TensorFlow](Background-TensorFlow.md) (Requires a CPU w/ AVX support).
+[PyTorch](Background-PyTorch.md) (Requires a CPU w/ AVX support).
 
 #### Advanced: Local Installation for Development
 
@@ -148,6 +159,7 @@ this, you will need to install `mlagents` and `mlagents_envs` separately. From
 the repository's root directory, run:
 
 ```sh
+pip3 install torch -f https://download.pytorch.org/whl/torch_stable.html
 pip3 install -e ./ml-agents-envs
 pip3 install -e ./ml-agents
 ```

docs/Learning-Environment-Executable.md

@@ -171,7 +171,7 @@ INFO:mlagents.trainers: first-run-0: Ball3DLearning: Step: 10000. Mean Reward: 2
 ```
 
 You can press Ctrl+C to stop the training, and your trained model will be at
-`results/<run-identifier>/<behavior_name>.nn`, which corresponds to your model's
+`results/<run-identifier>/<behavior_name>.onnx`, which corresponds to your model's
 latest checkpoint. (**Note:** There is a known bug on Windows that causes the
 saving of the model to fail when you early terminate the training, it's
 recommended to wait until Step has reached the max_steps parameter you set in
@@ -182,6 +182,6 @@ following the steps below:
    `Project/Assets/ML-Agents/Examples/3DBall/TFModels/`.
 1. Open the Unity Editor, and select the **3DBall** scene as described above.
 1. Select the **3DBall** prefab from the Project window and select **Agent**.
-1. Drag the `<behavior_name>.nn` file from the Project window of the Editor to
+1. Drag the `<behavior_name>.onnx` file from the Project window of the Editor to
    the **Model** placeholder in the **Ball3DAgent** inspector window.
 1. Press the **Play** button at the top of the Editor.

docs/ML-Agents-Overview.md

@@ -35,7 +35,7 @@ open-source project that enables games and simulations to serve as environments
 for training intelligent agents. Agents can be trained using reinforcement
 learning, imitation learning, neuroevolution, or other machine learning methods
 through a simple-to-use Python API. We also provide implementations (based on
-TensorFlow) of state-of-the-art algorithms to enable game developers and
+PyTorch) of state-of-the-art algorithms to enable game developers and
 hobbyists to easily train intelligent agents for 2D, 3D and VR/AR games. These
 trained agents can be used for multiple purposes, including controlling NPC
 behavior (in a variety of settings such as multi-agent and adversarial),
@@ -51,9 +51,9 @@ transition to the ML-Agents Toolkit easier, we provide several background pages
 that include overviews and helpful resources on the
 [Unity Engine](Background-Unity.md),
 [machine learning](Background-Machine-Learning.md) and
-[TensorFlow](Background-TensorFlow.md). We **strongly** recommend browsing the
+[PyTorch](Background-PyTorch.md). We **strongly** recommend browsing the
 relevant background pages if you're not familiar with a Unity scene, basic
-machine learning concepts or have not previously heard of TensorFlow.
+machine learning concepts or have not previously heard of PyTorch.
 
 The remainder of this page contains a deep dive into ML-Agents, its key
 components, different training modes and scenarios. By the end of it, you should
@@ -280,7 +280,7 @@ for additional information.
 
 ### Custom Training and Inference
 
-In the previous mode, the Agents were used for training to generate a TensorFlow
+In the previous mode, the Agents were used for training to generate a PyTorch
 model that the Agents can later use. However, any user of the ML-Agents Toolkit
 can leverage their own algorithms for training. In this case, the behaviors of
 all the Agents in the scene will be controlled within Python. You can even turn

docs/Readme.md

@@ -11,7 +11,7 @@
 - [ML-Agents Toolkit Overview](ML-Agents-Overview.md)
   - [Background: Unity](Background-Unity.md)
   - [Background: Machine Learning](Background-Machine-Learning.md)
-  - [Background: TensorFlow](Background-TensorFlow.md)
+  - [Background: PyTorch](Background-PyTorch.md)
 - [Example Environments](Learning-Environment-Examples.md)
 
 ## Creating Learning Environments