What is the Vision-to-Teach Technology?

1. Scanning the Object with the Robot’s Camera

The process starts when the robot uses its built-in camera system to scan the target object. It captures high-resolution images and possibly other sensory data to build a detailed digital representation. This first scan collects key information about the object’s size, shape, texture, and other physical properties—information the robot needs for accurate interaction.

2. Placing the Object on a Surface

After the initial scan, the object is set on a specific surface in the robot’s operating environment. By positioning the object in this way, the robot can understand how it sits relative to its surroundings. This step is essential for tasks that require picking up or manipulating the object from a particular location.

3. Secondary Scanning by the Robot

Once the object is in place, the robot performs a secondary scan to refine its understanding of the object’s location and orientation. This scan helps ensure that any changes in position are reflected in the robot’s view, providing real-time data for precise interaction.

4. Demonstration Through Hand Interaction and 3D Data Recording

A human operator demonstrates the task by interacting with the object manually—such as picking it up, moving it, or carrying out a specific manipulation. During the demonstration, the robot records everything with its camera system. Advanced software then converts the recorded visual information into three-dimensional (3D) data models, capturing details of the movement, including hand positioning and motion trajectories. The resulting 3D data acts as a detailed template for the robot to learn the task.

5. Utilizing Data in the Omniverse for 3D Scene Generation and Robot Path Practice

The collected 3D data is imported into a virtual simulation environment, often called the “Omniverse.” In this virtual space, a 3D scene is created that mirrors the real-world environment and the demonstrated task. The robot then practices the task repeatedly in simulation, improving its movements and interactions. Machine learning algorithms evaluate the robot’s performance, enabling it to optimize its path and actions. The robot continues this iterative process in the virtual environment until it can accurately reproduce the demonstrated movement, including grasping and manipulating the object as intended.

6. Transferring the Trained Digital Twin Model to the Robot for Execution

After the robot has successfully learned and practiced the task in the virtual environment, the trained digital twin model—containing all the learned behaviors and movement patterns—is transferred back to the physical robot. This updated model provides the instructions the robot needs to perform the task in the real world. The robot then carries out the movement autonomously, using the exact motions and interactions it learned during virtual simulations.

Conclusion

The Visio-to-Teach technology simplifies robot programming by replacing traditional coding with visual demonstrations and simulation-based learning. By observing human actions and practicing in a virtual environment, robots can learn complex tasks quickly with high precision. This technology also improves a robot’s adaptability and efficiency, making it a strong solution for industries that rely on robotic automation.