Dextra/pose estimation

README.md

@@ -10,15 +10,28 @@ This is a central repository for tools, tutorials, resources, and documentation
 
 Simulation plays an important role in robotics development, and we’re here to ensure that roboticists can use Unity for these simulations. We're starting off with a set of tools to make it easier to use Unity with existing ROS-based workflows. Try out some of our samples below to get started quickly.
 
-## Getting Started with Unity Robotics
+## Getting Started
+### [Quick Installation Instructions](tutorials/quick_setup.md)
+
+Brief steps on installing the Unity Robotics packages.
+
+### [Pick-and-Place Demo](tutorials/pick_and_place/README.md)
+
+A complete end-to-end demonstration, including how to set up the Unity environment, how to import a robot from URDF, and how to set up two-way communication with ROS for control.
+
+### [**New**][Pose Estimation Demo](tutorials/pose_estimation/README.md)
+
+A complete end-to-end demonstration, in which we collect training data in Unity, and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.
+
+### [Articulations Robot Demo](https://github.com/Unity-Technologies/articulations-robot-demo)
+
+A robot simulation demonstrating Unity's new physics solver (no ROS dependency).
+## Documentation
 
 | Tutorial | Description |
 |---|---|
-| [Quick Installation Instructions](tutorials/quick_setup.md) | Brief steps on installing the Unity Robotics packages |
-| [Pick-and-Place Demo](tutorials/pick_and_place/README.md) | A complete end-to-end demonstration, including how to set up the Unity environment, how to import a robot from URDF, and how to set up two-way communication with ROS for control |
 | [ROS–Unity Integration](tutorials/ros_unity_integration/README.md) | A set of component-level tutorials showing how to set up communication between ROS and Unity |
 | [URDF Importer](tutorials/urdf_importer/urdf_tutorial.md) | Steps on using the Unity package for loading [URDF](http://wiki.ros.org/urdf) files | 
-| [Articulations Robot Demo](https://github.com/Unity-Technologies/articulations-robot-demo) | A robot simulation demonstrating Unity's new physics solver (no ROS dependency)
 
 
 ## Component Repos
@@ -66,4 +79,4 @@ For general questions, feedback, or feature requests, connect directly with the
 For bugs or other issues, please file a GitHub issue and the Robotics team will investigate the issue as soon as possible.
 
 ## License
-[Apache License 2.0](LICENSE)
+[Apache License 2.0](LICENSE)

tutorials/pose_estimation/.gitignore

@@ -0,0 +1,7 @@
+*.DS_Store
+*.vsconfig
+ROS/src/robotiq
+ROS/src/universal_robot
+ROS/src/ros_tcp_endpoint
+ROS/src/moveit_msgs
+ROS/src/ur3_moveit/models/

tutorials/pose_estimation/Documentation/0_ros_setup.md

@@ -0,0 +1,67 @@
+# Pose-Estimation-Demo Tutorial: Part 0
+
+This page provides steps on how to manually set up a catkin workspace for the Pose Estimation tutorial.
+
+1. Navigate to the `pose_estimation/` directory of this downloaded repository. This directory will be used as the ROS catkin workspace.
+
+2. Copy or download this directory to your ROS operating system if you are doing ROS operations in another machine, VM, or container.
+
+3. If the following packages are not already installed on your ROS machine, run the following commands to install them:
+
+```bash
+sudo apt-get update && sudo apt-get upgrade
+sudo apt-get install python3-pip ros-noetic-robot-state-publisher ros-noetic-moveit ros-noetic-rosbridge-suite ros-noetic-joy ros-noetic-ros-control ros-noetic-ros-controllers ros-noetic-tf* ros-noetic-gazebo-ros-pkgs ros-noetic-joint-state-publisher
+sudo pip3 install rospkg numpy jsonpickle scipy easydict torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
+```
+
+> Note: If you encounter errors installing Pytorch via the above `pip3` command, try the following instead:
+> ```bash 
+> sudo pip3 install rospkg numpy jsonpickle scipy easydict torch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
+> ```
+
+
+Most of the ROS setup has been provided via the `ur3_moveit` package. This section will describe the provided files.
+
+4. If you have not already built and sourced the ROS workspace since importing the new ROS packages, navigate to your ROS workplace, and run: 
+
+```bash 
+catkin_make
+source devel/setup.bash
+```
+
+Ensure there are no unexpected errors.
+
+The ROS parameters will need to be set to your configuration in order to allow the server endpoint to fetch values for the TCP connection. 
+
+5. Navigate to your ROS workspace (e.g. `~/catkin_ws`). Assign the ROS IP in the `params.yaml` file as follows:
+
+```bash
+echo "ROS_IP: $(hostname -I)" > src/ur3_moveit/config/params.yaml
+```
+
+>Note: You can also manually assign this value by navigating to the `src/ur3_moveit/config/params.yaml` file and opening it for editing.
+>```yaml
+>ROS_IP: <your ROS IP>
+>```
+>e.g.
+>```yaml
+>ROS_IP: 10.0.0.250
+>```
+
+> Note: Learn more about the server endpoint and ROS parameters [here](https://github.com/Unity-Technologies/Unity-Robotics-Hub/blob/main/tutorials/ros_unity_integration/server_endpoint.md).
+
+This YAML file is a rosparam set from the launch files provided for this tutorial, which has been copied below for reference. Additionally, the `server_endpoint`, `pose estimation`, and `mover` nodes are launched from this file.
+
+```xml
+<launch>
+    <rosparam file="$(find ur3_moveit)/config/params.yaml" command="load"/>
+    '<include file="$(find ur3_moveit)/launch/demo.launch" />
+    <node name="server_endpoint" pkg="ur3_moveit" type="server_endpoint.py" args="--wait" output="screen" respawn="true" />
+    <node name="pose_estimation" pkg="ur3_moveit" type="pose_estimation_script.py" args="--wait" output="screen"/>
+    <node name="mover" pkg="ur3_moveit" type="mover.py" args="--wait" output="screen" respawn="true" respawn_delay="2.0"/>
+</launch>
+```
+
+>Note: The launch files for this project are available in the package's launch directory, i.e. `src/ur3_moveit/launch/`.
+
+The ROS workspace is now ready to accept commands! Return to [Part 4: Set up the Unity side](4_pick_and_place.md#step-3) to continue the tutorial.

tutorials/pose_estimation/Documentation/1_set_up_the_scene.md

@@ -0,0 +1,189 @@
+# Pose Estimation Demo: Part 1
+
+In this first part of the tutorial, we will start by downloading and installing the Unity Editor. We will install our project's dependencies: the Perception, URDF, and TCP Connector packages. We will then use a set of provided prefabs to easily prepare a simulated environment containing a table, a cube, and a working robot arm. 
+
+
+**Table of Contents**
+  - [Requirements](#reqs)
+  - [Create a New Project](#step-1)
+  - [Download the Perception, the URDF and the TCP connector Packages](#step-2)
+  - [Setup the Ground Truth Render Feature](#step-3)
+  - [Setup the Scene](#step-4)
+
+---
+
+### <a name="reqs">Requirements</a>
+
+To follow this tutorial you need to **clone** this repository even if you want to create your Unity project from scratch.
+
+
+  > Note For Windows users: You need to have a software enabling you to run bash files. One option is to download [GIT](https://git-scm.com/downloads). During installation of GIT, add GIT Bash to windows context menu by selecting its option. After installation, right click in your folder, and select [GIT Bash Here](Images/0_GIT_installed.png).
+
+1. Open a terminal and put yourself where you want to host the repository. 
+```bash
+git clone https://github.com/Unity-Technologies/Unity-Robotics-Hub.git
+```
+
+Then we need to be in the `Unity-Robotics-Hub/tutorials/pose_estimation` folder and generate the contents of the `universal_robot`, `moveit_msgs`, `ros_tcp_endpoint`, and the `robotiq` folders. 
+```bash
+cd Unity-Robotics-Hub/tutorials/pose_estimation
+./submodule.sh
+```
+
+2. [Install Unity `2020.2.*`.](install_unity.md)
+
+
+### <a name="step-1">Create a New Project</a> 
+When you first run Unity, you will be asked to open an existing project, or create a new one.
+
+1. Open Unity and create a new project using the **Universal Render Pipeline**. Name your new project _**Pose Estimation Tutorial**_, and specify a desired location as shown below.
+
+<p align="center">
+<img src="Images/1_create_new_project.png" align="center" width=950/>
+</p>
+
+### <a name="step-2">Download the Perception, the URDF and the TCP connector Packages</a>
+
+Once your new project is created and loaded, you will be presented with the Unity Editor interface. From this point on, whenever we refer to the "editor", we mean the Unity Editor.
+
+#### How to install packages
+We will need to download and install several packages. In general, packages can be installed in Unity with the following steps:
+
+- From the top menu bar, open _**Window**_ -> _**Package Manager**_. As the name suggests, the _**Package Manager**_ is where you can download new packages, update or remove existing ones, and access a variety of information and additional actions for each package.
+
+- Click on the _**+**_ sign at the top-left corner of the _**Package Manager**_ window and then choose the option _**Add package from git URL...**_. 
+
+- Enter the package address and click _**Add**_. It will take some time for the manager to download and import the package.
+
+Installing the different packages may take some time (few minutes). 
+
+<p align="center">
+<img src="Gifs/1_package_imports.gif"/>
+</p>
+
+
+#### Install Dependencies
+Install the following packages with the provided git URLs:
+
+1. [Perception package](https://github.com/Unity-Technologies/com.unity.perception) - `[email protected]`
+    * This will help us collect training data for our machine learning model. 
+
+2. [URDF Importer package](https://github.com/Unity-Technologies/URDF-Importer) - `https://github.com/Unity-Technologies/URDF-Importer.git#v0.1.2`
+    * This package will help us import a robot into our scene from a file in the [Unified Robot Description Format (URDF)](http://wiki.ros.org/urdf).
+
+3. [TCP Connector package](https://github.com/Unity-Technologies/ROS-TCP-Connector) - `https://github.com/Unity-Technologies/ROS-TCP-Connector.git#v0.1.2`
+    * This package will enable a connection between ROS and Unity. 
+
+>Note: If you encounter a Package Manager issue, check the [Troubleshooting Guide](troubleshooting.md) for potential solutions.
+
+### <a name="step-3">Set up Ground Truth Render Feature</a>
+
+The Hierarchy, Scene View, Game View, Play/Pause/Step toolbar, Inspector, Project, and Console windows of the Unity Editor have been highlighted below for reference, based on the default layout. Custom Unity Editor layouts may vary slightly. A top menu bar option is available to re-open any of these windows: Window > General.
+
+<p align="center">
+<img src="Images/1_scene_overview.png"/>
+</p>
+
+
+The perception packages relies on a "ground truth render feature" to save out labeled images as training data. You don't need to worry about the details, but follow the steps below to add this component:
+
+1. The _**Project**_ tab contains a search bar; use it to find the file named `ForwardRenderer`, and click on the file named `ForwardRenderer.asset` as shown below:
+
+<p align="center">
+<img src="Images/1_forward_renderer.png"/>
+</p>
+
+2. Click on the found file to select it. Then, from the _**Inspector**_ tab of the editor, click on the _**Add Renderer Feature**_ button, and select _**Ground Truth Renderer Feature**_ from the dropdown menu:
+
+<p align="center">
+<img src="Images/1_forward_renderer_inspector.png" width=430 height=350/>
+</p>
+
+
+### <a name="step-4">Set up the Scene</a>
+
+#### The Scene
+Simply put in Unity, a `Scene` contains any object that exists in the world. This world can be a game, or in this case, a data-collection-oriented simulation. Every new project contains a Scene named SampleScene, which is automatically opened when the project is created. This Scene comes with several objects and settings that we do not need, so let's create a new one.
+
+1. In the _**Project**_ tab, right-click on the `Assets > Scenes` folder and click _**Create -> Scene**_. Name this new Scene `TutorialPoseEstimation` and double-click on it to open it. 
+
+The _**Hierarchy**_ tab of the editor displays all the Scenes currently loaded, and all the objects currently present in each loaded Scene, as shown below:
+<p align="center">
+<img src="Images/1_hierarchy.png" width=500 height=200/>
+</p>
+
+As seen above, the new Scene already contains a camera (`Main Camera`) and a light (`Directional Light`). We will now modify the camera's field of view and position to prepare it for the tutorial.
+
+2. Still in the _**Inspector**_ tab of the `Main Camera`, modify the camera's `Position` and `Rotation` to match the values shown below. This orients the camera so that it will have a good view of the objects we are about to add to the scene.   
+
+<p align="center">
+<img src="Images/1_camera_settings.png" height=117 width=500/>
+</p>
+
+3. Click on `Directional Light` and in the _**Inspector**_ tab, modify the light's `Position` and `Rotation` to match the screenshot below. 
+
+<p align="center">
+<img src="Images/1_directional_light.png" height=217 width=500/>
+</p>
+
+#### Adding Tutorial Files
+Now it is time to add some more objects to our scene. Before doing so, we need to import some folders containing the required assets. 
+
+4. Download [TutorialAssets.zip](https://github.com/Unity-Technologies/Unity-Robotics-Hub/releases/download/Pose-Estimation/TutorialAssets.zip), and unzip it. It should contain the following subfolders: `Materials`, `Prefabs`, `RosMessages`, `Scripts`, `URDFs`.
+
+5. Drag and Drop the `TutorialAssets` folder onto the `Assets` folder in the _**Project**_ tab. 
+
+Your `Assets` folder should like this: 
+
+<p align="center">
+<img src="Images/1_assets_preview.png"/>
+</p>
+
+#### Using Prefabs
+Unity’s [Prefab](https://docs.unity3d.com/Manual/Prefabs.html) system allows you to create, configure, and store a **GameObject** complete with all its components, property values, and child **GameObjects** as a reusable **Asset**. It is a convenient way to store complex objects. 
+
+A prefab is just a file, and you can easily create an instance of the object in the scene from a prefab by dragging it into the _**Hierarchy panel**_.
+
+For your convenience, we have provided prefabs for most of the components of the scene (the cube, goal, table, and floor).
+
+6. In the _**Project**_ tab, go to `Assets > TutorialAssets > Prefabs > Part1` and drag and drop the `Cube` prefab inside the _**Hierarchy panel**_.
+
+7. Repeat the action with the `Goal`, `Table` and the `Floor`. 
+
+
+<p align="center">
+<img src="Gifs/1_import_prefabs.gif"/>
+</p>
+
+>Note: If you encounter an issue with the imported prefab materials, check the [Troubleshooting Guide](troubleshooting.md) for potential solutions.
+
+
+#### Importing the Robot
+Finally we will add the robot and the URDF files in order to import the UR3 Robot. 
+
+8. In the _**Project**_ tab, go to `Assets > TutorialAssets > URDFs > ur3_with_gripper` and right click on the `ur3_with_gripper.urdf` file and select `Import Robot From Selected URDF file`. A window will pop up, keep the default **Y Axis** type in the Import menu and the **Mesh Decomposer** to `VHACD`. Then, click Import URDF. These set of actions are showed in the following video. 
+
+>Note Unity uses a left-handed coordinate system in which the y-axis points up. However, many robotics packages use a right-handed coordinate system in which the z-axis or x-axis points up. For this reason, it is important to pay attention to the coordinate system when importing URDF files or interfacing with other robotics software.
+
+>Note: VHACD algorithm produces higher quality convex hull for collision detection than the default algorithm.
+
+<p align="center">
+<img src="Gifs/1_URDF_importer.gif" width=800 height=548/>
+</p>
+
+>Note: If you encounter an issue with importing the robot, check the [Troubleshooting Guide](troubleshooting.md) for potential solutions.
+
+#### Setting up the Robot
+
+9. Select the `ur3_with_gripper` GameObject and in the _**Inspector**_ view, go to the `Controller` script and set the `Stiffness` to **10000**, the `Damping` to **1000** and the `Force Limit` to **1000**. These are physics properties that control how the robot moves.
+
+10. In the _**Hierarchy**_ tab, select the `ur3_with_gripper` GameObject and click on the arrow on the left, then click on the arrow on the left of `world`, then on `base_link`. In the `Articulation Body` component, toggle on `Immovable` for the `base link`. This will fix the robot base to its current position.
+
+<p align="center">
+<img src="Gifs/1_robot_settings.gif" width=800 height=465/>
+</p>
+
+
+### Proceed to [Part 2](2_set_up_the_data_collection_scene.md).
+
+