Imitation Learning
Overview
This document provides an overview of the complete ROS 2-based imitation learning pipeline built on the OMY and the Hugging Face Hub and features a user-friendly web GUI designed to streamline interaction and enhance accessibility.
1. Data Collection
Human operators use a wearable skeletal leader device to demonstrate motions, collecting image and joint position data. The web-based GUI plays a key role in streamlining the process by providing real-time visualization and control. The collected data can be uploaded to and downloaded from the Hugging Face Hub.
2. Data Visualization
Collected data is visualized to inspect motion trajectories and images, helping to identify potential errors prior to training.
3. Model Training
The verified dataset is then used to train an action policy model. Training can be performed on local GPUs or on embedded platforms such as the NVIDIA Jetson. The resulting model can be uploaded to and downloaded from the Hugging Face Hub.
4. Model Inference
Once trained, the models are deployed on the OMY to execute real-time inference for tasks such as picking, placing, and obstacle avoidance.
End-to-End Imitation Learning Workflow
- The diagram below shows the full imitation learning workflow using the OMY and Hugging Face.
Tutorial Videos: End-to-End Imitation Learning Workflow
The video illustrates the full imitation learning workflow using the OMY, including teleoperation, dataset creation, and real-time inference with a trained model:
- Full imitation learning workflow using OMY and the Web GUI.
- Dataset Schema
The dataset follows the standard 🤗 Hugging Face datasets format and contains imitation learning demonstrations collected from the OMY via ROS 2 teleoperation using the LeRobot framework.
| Field | Type | Description |
|---|---|---|
action | List[float32] | Leader state vector |
observation.state | List[float32] | Follower state vector |
observation.images.cam_head | Image | RGB image from the head-mounted camera |
observation.images.cam_wrist_1 | Image | RGB image from the first wrist camera |
observation.images.cam_wrist_2 | Image | RGB image from the second wrist camera |
timestamp | float32 | Time (in seconds) when the step was recorded |
frame_index | int64 | Index of the frame within an episode |
episode_index | int64 | Index of the episode |
index | int64 | Global index across the dataset |
task_index | int64 | Task identifier |